The CI/CD section focuses on the following stages of the DevOps Lifecycle:
You can see how the CI/CD stages relate to each other in the following infographic:
CI/CD is a large portion of the DevOps market with significant growth upside. The application release automation/CD and configuration management (mainly aligned to the Release stage) segment supported 984.7MM USD revenue in 2019, projected by 2022 to grow to 1617.0MM. The Continuous Integration, integrated quality/code review/static analysis/test automation, and environment management (mainly aligned to the Verify stage) represented an even larger 1785.0MM market in 2019, projected to grow to 2624.0MM. With a combined projected total of 4241.0MM by 2022, it's critical that we continue to lead here. (market data, GitLab internal link)
The CI/CD section has 41 members with 22 vacancies open. Tracking against plan can be seen on our hiring chart. Verify is at year 7 maturity, Package is at year 3 maturity, and Release is at year 4 maturity.
We have a monthly internal Q&A related to the content on this page and the team's way of working. You can see the latest video below:
CI/CD solutions have continued to rapidly innovate. Kubernetes itself, a massive driver of industry change, just turned five years old in 2019. Docker, now considered a mature and foundational technology, was only released in 2013. We do not expect this pace to slow down, so it's important we balance supporting the current technology winners (Kubernetes, AWS, Azure, and Google Cloud) with best-of-breed, built-in solutions, while at the same time avoiding over-investment on any single technology solution that could become legacy.
There are dangers to falling out of balance; Pivotal resisted embracing Kubernetes early and has suffered for it. At the same time, technologies like Jenkins that deeply embraced what are now considered legacy paradigms are enjoying the long tail of relevancy, but are having to scramble to remain relevant for the future.
Continuing apace in 2019 after Microsoft's 2018 acquisition of GitHub, consolidation as a trend seems here to stay. In January, Travis CI was acquired by Idera, and in February we saw Shippable acquired by JFrog. Atlassian and GitHub now both bundle CI/CD with SCM, alongside their ever-growing related suite of products.
It's natural for technology markets go through stages as they mature: when a young technology is first becoming popular, there is an explosion of tools to support it. New technologies have rough edges that make them difficult to use, and early tools tend to center around adoption of the new paradigm. Once the technology matures, consolidation is a natural part of the life cycle. GitLab is in a fantastic position to be ahead of the curve on consolidation, but it's a position we need to actively defend as competitors start to bring more legitimately integrated products to market.
Our product positioning for CI/CD is aligned to the one set forth in Jez Humble's classic Continuous Delivery book. Reducing risk, getting features and fixes out quickly, and reducing costs/creating happier teams by removing the barriers to getting things done are principles that have stood the test of time. Modern CD pushes this even further by supporting more ways to microtarget users and incrementally make changes, and Progressive Delivery has emerged as a set of best-practices that we're embracing in our product. Still, the 5 principles at the heart of continuous delivery remain illustrative of the approaches we want to enable:
One of the challenges with CD today is that it requires a suite of different tools to deliver: feature flag management, deployment automation, release orchestration, and more. GitLab's single application is uniquely positioned to provide a compelling solution here.
The strategy for CI/CD overall centers on the idea of enabling workflows that support teams interacting with the same concepts and tools around CI/CD; each may have their own views or dashboards in the product that support their day to day, but the information about the state of what is deployed where is available everywhere. For example, a person thinking about upcoming releases may interact mostly with an environment-based overview that helps them see upcoming changes as they flow through different ephemeral or permanent environments, but that state data exists everywhere it is relevant:
The end result of this kind of information flow is that even complex delivery flows become part of everyone's primary way of working. There isn't a context shift (or even worse, a switch into a different tool) needed to start thinking about delivery - that information is there, in context, from your first commit.
Given our vision and strategy, our plan for the next year has been organized around a number of themes that support our direction:
DevOps tooling is currently heading in the cycle towards consolidation/bundling, simply because users are exhausted from the administrative effort needed to wire individual solutions together in order to realize the benefits of a well-integrated experience. Point application solutions like TravisCI and CircleCI are struggling at this point because they suffer from the fundamental problem that CI/CD is truly not an island: in order to make sure their solutions can compete effectively they are forced to maintain integrations with various stacks, which is expensive in terms of focus and results that they are unable to innovate as fast as competitors that are providing a unified solution. It's important we retain flexibility for integrations if unbundling becomes a trend or a high quality point solution emerges within any segment, but overall right now it's clear that our current approach is winning.
In the end, our single application solution is what will help us ensure that GitLab is a Robot Cat instead of a Sheepdog. By embedding CI/CD naturally into everyone's workflows, as a first-class part of working with issues, source control, merge requests, etc., CI/CD isn't a box under someone's desk that nobody knows how to work while on vacation; it's a part of how everyone works together - the glue that binds everyone around how they deliver software together as a team, providing unified metrics all the way from ideation to delivery.
GitLab has traditionally done very well with building and deploying software on Linux, with our shared runner fleet making it very easy to get up and running quickly. We've supported MacOS and Windows Runners, but only in a bring-your-own configuration where you host and register your own Runners. Our top priority is changing that by improving Runner features for these platforms in addition to providing them as available options in the shared runner fleet. Just being able to run builds on the platform isn't sufficient, though. We also want to improve our support for unit testing and code coverage frameworks used on these operating systems, as well as our support for different package managers that are used by code build there.
Truly being multi-platform also means making sure that GitLab interacts well with other platforms and tools you use as part of your DevOps ecosystem. One important element of this for CI/CD in particular is making sure that GitLab CI/CD remains a viable, performant option even when using external repositories.
We recently released the custom executor which allows for more kinds of platforms to be supported in the GitLab Runner. This can include GPUs, TPUs, FPGAs, RISC-V, ARM, and more. We also want to start offering multiple sizes for shared runners, so you can match large jobs with powerful runners. We are also looking at making multi-platform runner minutes available for self-managed to purchase, and building out a Runner for Fargate.
At the core, GitLab CI/CD was built from the ground up with speed and scalability in mind. This is reflected both in the product design and product vision (automated testing, testing all the things) To enable our users to go from monthly shipping releases to truly enabling continuous delivery, everything that we do must be concerned with the scalability and speed of continuous integration and testing. Not that sword fighting on office chairs isn't fun, but we do want people to be able to contribute at the speed of development.
Faster feedback is clearly the future of Continuous Integration solutions. We believe we can best serve developers by leveraging the integrated nature of GitLab to further shift left checks that would normally happen when a pipeline runs. Examples of opportunities for faster feedback include but are not limited to spell check, dependency checks and linting while code creation is happening in the Web IDE.
The time it takes to set up the execution agent for every job, before the first line of your script is run, is also a critical measure for a good CI solution. This can include fetching the image, provisioning a host, running internal setup, polling from the runner, and so on; each of these add delays to every job, but do not contribute to actually doing the important work. To track and understand how we measure this, see the time to script execution epic. This also includes elements of improving the way the fleet of shared runners works, such as re-enabling registry mirroring (infrastructure#3158), and doing things like pre-creating the runner environment for upcoming jobs
Wall clock time for a pipeline is also an interesting measure. This is the one you really feel - when you make a commit and a pipeline starts, and you need to go have lunch before you see the result, you're really suffering. Our vision here is to have GitLab help you identify the bottlenecks in your pipelines and ideally remediate them automatically, but at least make it visibile where the bottlenecks are so you can investigate. Issues like our auto-determining the test parallelization factor and running tests likely to fail first represent how we want GitLab CI/CD to automatically find and solve these problems for you. We are building more speed into the workflow itself with features like the dependency proxy.
One of the core principles of GitLab CI/CD is the idea that we want to give people powerful tools to do their job; flexibility is key, and we want to ensure that getting complex things done with small, easy-to-use, but powerful primitives is the GitLab way. This also helps us remain flexible and avoid hewing too closely to a specific technology solution, ensuring we continue to diverse and sometimes legacy technology stacks, and retain our flexibility to embrace future innovation. Put simply, we will always prioritize powerful building blocks in GitLab that are exposed to you, the user. Where we have structures to get started quickly with a given stack, those will be built up from the same primitives that you have access to as well.
An interesting measure for this is how long it takes to get to your first green build from the time you create your project, for both new and seasoned pipeline users. We are going to start tracking this as a success metric for how easy it is to get in and get your work done; GitLab's built-in functionality should make this quick and easy.
Put simply, Progressive Delivery is a set of emerging CD best practices, oriented around being able to control and monitor deployments incrementally over time, and in an automated and safe way.
Progressive delivery controls the deployment to a specific subset of users before being fully shipped, so that testing in production doesn’t have to be a massive risk. The idea is to get feedback, and fast. Similarly to Compliance as Code, our single application is uniquely capable of providing an integrated solution in this space by using feature flags, incremental rollout and other traffic routing methods that can be controlled directly on the deployment pipeline. You can control which users and what features can be accessed in addition to where and when, giving you ultimate control of your production environments and providing you the ability to make quick decisions when it matters. Deploy boards and feature flags dashboards will give you additional insights into the deployment rollout and let you determine it's audience and progress. They will even offer recommendations when the deployment can be considered complete and a feature flag may longer be needed.
Delivering software to your environments automatically is the first step, and modern technologies like Kubernetes are now allowing on-demand environments to be created like never before. By tying your source repo to intelligent infrastructure that understands the intentions of developers, you can enable a true one-click workflow to deliver to your internal or external end users.
Mobile is an important segment of the market - many consumer brands, even those that would be considered non-tech are now development and distributing mobile apps as part of their customer engagement strategies. It's important for GitLab that we support these customers to be successful in this area. To see an example of where we're headed and the kinds of easy-to-use delivery flows we have in mind, we have a video where we demonstrate (using GitLab today) making a change to a mobile app using an iPad Pro, sending the build over to a MacOS runner, and receive a working copy back to test on the very same device using TestFlight. Part of supporting Mobile builds also includes our vision for better multi-platform support, making shared runners available for MacOS in particular will allow for easier iOS builds on GitLab.
More information on how we plan to support mobile CI/CD can be found on our mobile use case page.
Compliance as Code is the idea that requirements for auditing and compliance teams can be collected with zero additional cognitive load from developers and other team members participating in software delivery. Because GitLab has an end-to-end view of the artifacts that are associated with releases (issues, merge requests, feature flags, and so on) we are uniquely positioned to provide a comprehensive view of compliance-related activity, collected on an ongoing basis throughout the build, test, and release process. Our vision here begins by improving our releases feature to start collecting evidence from the cycle leading up to the release event (gitlab#26019).
In addition to collecting compliance evidence along the path to production, our plan is to improve the way GitLab pipelines interact with secure secrets, typically held in a Vault or other secrets store. Our plan is to first allow GitLab to integrate with your existing Vault, implemented via gitlab#28321. In the future, however, we have a broader vision for bundling Vault with GitLab and making secrets management a part of the GitLab product in general; this includes storing GitLab's own secrets in an external Vault.
GitOps is an emerging concept that wraps the entirety of software delivery into a git-first workflow, a lot of which GitLab actually already supports:
One proposed aspect of GitOps though, apart from keeping Git as the source of truth and doing operations by Merge Request which we have embraced, is inverting the deployment model to be from push (where jobs execute and push changes to an environment, which can then be monitored for divergence) to pull (where the environment itself runs an operator that monitors Git and itself, ensuring that everything stays in sync.) There are potential control benefits here, in that the CI/CD system does not need to maintain access to the target environment, but the reality is is that the CI/CD system will continue to be able to influence the environment in various ways by changing state in Git. Additionally, bringing the state change automation inside the cluster brings benefits to observability by seeing the environment and the state change in a single place.
For us, the benefits here are less clear than the core GitOps principles of everything-as-code, everything in Git, and change via Merge Request. We'll continue to monitor how this area evolves, and potentially change course as more innovation or clear benefits to this aspect are realized, but in the meantime we are not investing in this area.
The CI/CD section is composed of three stages, each of which contains several categories. Each stage has an overall strategy statement below, aligned to the themes for CI/CD, and each category within each stage has a dedicated vision page (plus optional documentation, marketing pages, and other materials linked below.)
The long term goals of the Verify group are:
To accomplish these goals we will focus on the themes and opportunities of mutli-platform support, Machine Learning , CI/CD analytics and test run parallelization.
Users of Verify's CI solution can expect us to invest heavily in multi-platform support, making it just as easy to build on Windows and MacOS as it is to build on Linux today. In terms of pipeline capabilities, we've recently implemented directed acyclic graphs, a powerful primitive for complex dependency relationships, and we will expand on this with great features like child/parent pipelines and cartesian products for jobs. The innovation never really stops here: there's always a way to provide a new, powerful command that can help organize your pipeline configuration in a simpler and more expressive way. Our mission is to make GitLab CI the easiest CI system to get up and running with, that also is the most powerful.
There is also an opportunity for us to take advantage of emerging AI and ML technologies to provide leverage to customers storing and building their source code within GitLab. We can see opportunities like dynamically modifying the order of unit tests to run the most likely to fail first. You can even imagine a future where GitLab offers suggestions of code improvements thereby accelerating the payment of tech debt and allowing the team to solve more problems that deliver value. While these examples are specific to Unit Testing we think this approach can apply to any of the testing categories.
Beyond pipeline execution, we'll improve how we present and act on interesting data around what jobs do and how they perform; timing and efficiency information is one part, but so is how test results are changing over time. Our epic for CI/CD analytics captures some of the important metrics we've identified so far. This data then can be acted upon by GitLab itself to automatically tune your pipelines for better performance - imagine automatically tuning parallelization of your test runs - without users having to spend time thinking about it. Self-healing pipelines that automatically adapt to changes to maintain their efficiency will be a powerful tool for teams.
The Verify stage is made up of several categories:
Gain the confidence to ship at blistering speed and immense scale with automated builds, testing, and out-of-the-box security to verify each commit moves you forward. This category is at the "lovable" level of maturity.
GitLab Runner is the execution agent that is used to run your CI jobs and send the results back to GitLab. This category is planned, but not yet available.
Automatically analyze your source code to surface issues and see if quality is improving or getting worse with the latest commit. This category is at the "minimal" level of maturity.
Unit testing ensures that individual components built within a pipeline perform as expected, and are an important part of a Continuous Integration framework. This category is at the "minimal" level of maturity.
It's important to validate that the components of your system work together well. By performing integration tests as part of your CI pipeline, you help ensure quality in every build. This category is planned, but not yet available.
Be confident in the performance of your changes by ensuring that they are validated against real world load scenarios. This category is planned, but not yet available.
Ensure performance of your software in-browser using automated web performance testing. This category is at the "minimal" level of maturity.
Modern software is often delivered as a collection of (micro)services to multiple clouds, rather than a single monolith to your own data center. Validating complex interactions to ensure reliability of the system as a whole is more important than ever. This category is planned, but not yet available.
Testing to ensure usability flows work and are actually understandable and valuable to your users. This category is at the "minimal" level of maturity.
Beyond being a compliance requirement in many cases, accessibility testing is the right thing to do. This category is planned, but not yet available.
The goal of the Package group is to build a set of features that, within three years, will allow ninety percent of our customers to store all of their packages in GitLab. In order to accomplish this goal, we must:
Our immediate focus is on improving the usability of our existing registries and standardizing authentication and permissions for each package manager. This will help us by providing a consistent and delightful user experience across integrations, which will enable us to quickly add support for the most common package manager formats. It's important to note that since we will prioritize only the most commonly used package managers, we will rely on the GitLab Community to add support for the less common formats. In order to help drive communinity contributions, we will provide clear documentation and guidance for how add new package manager formats to GitLab.
In order to provide a seamless experience for building, testing and releasing code, we must create visibility and transparency into the Package and Container Registry. We know that the primary reasons users navigate to their registries are to figure out which image or package to use, confirm that an image or package was built or to troubleshoot a build that failed. Armed with that knowledge, we will ensure that the Registry provides meta data and evidence for how images/packages are built, by whom and when.
By accomplishing the above, we will help our customers to drive innersourcing. In other words, we will ensure that organizations are able to collaborate across teams and contribute shared packages and images. This will reduce risk by ensuring problems are found and fixed before the wrong person discovers them, reduce cost and inefficiencies by ensuring developers do not have to start from scratch, and improve conversational development by ensuring everyone can contribute.
Finally, we will work to improve the performance and reliability of builds by rolling out the dependency proxy to each package manager. Increased adoption of the dependency proxy will allow us to build new features that will help our customers to reduce open source risk by providing visibility and transparency into external dependencies.
The Package stage is made up of several categories:
Every team needs a place to store their packages and dependencies. GitLab aims to provide a comprehensive solution, integrated into our single application, that supports package management for all commonly used languages and binary formats. This category is at the "minimal" level of maturity.
A secure and private registry for Docker images built-in to GitLab. Creating, pushing, and retrieving images works out of the box with GitLab CI/CD. This category is at the "viable" level of maturity.
Kubernetes cluster integrations can take advantage of Helm charts to standardize their distribution and install processes. Supporting a built-in helm chart registry allows for better, self-managed container orchestration. This category is planned, but not yet available.
The GitLab dependency proxy can serve as an intermediary between your local developers and automation and the world of packages that need to be fetched from remote repositories. By adding a security and validation layer to a caching proxy, you can ensure reliability, accuracy, and auditability for the packages you depend on. This category is at the "minimal" level of maturity.
GitLab ensures that the dependencies stored in your package registries conform to your corporate compliance guidelines. This means you can prevent your organization from using dependencies that are insecure or out of policy. This category is planned, but not yet available.
The traditional roles of software development teams are changing. The trend in the market is to build self-managed teams and we believe that more and more teams will adopt this approach in the next three years. In particular, we have seen the traditional centralized Release Management role at companies change, becoming more of a shared responsibility that is distributed amongst several groups, and expect that to continue. We want to give these teams with these newfound responsibilities everything they need to be successful.
Our vision for Release at GitLab is that common deployment targets just work - point us to your code and your environment, and we'll automatically connect the dots. We do this well already for Kubernetes, but we are going to expand this to other targets such as Azure, Google Cloud Platform, and AWS, and also mobile marketplaces and other app stores. This deployment can be automatically tracked, monitored, and (if there are issues), rolled back.
To support this transition, we are bringing a variety of Release Management features into the product: release orchestration, runbooks for releases, management of blackout periods, environment scheduling, and more, but in a way where these features are integrated directly into the single application. For example, in addition to having a Release Manager view as a dashboard within the product, feature flag states will be connected with the issues and merge requests that introduced them. Releases will automatically collect evidence about what the state of issues, test results, permissions, and so on were at the time of the release automatically. This integrated way of working serves both the teams writing the software and the teams delivering it, and is a unique approach in the market compared to point solutions which often provide tooling usable for Release Managers, but do not integrate well with the rest of the team.
In addition to the change in roles, the next natural step of software delivery is Progressive Delivery. Once there is an automated CI/CD process in place, Progressive delivery techniques enable the deployment to a targeted subset of users before being released to everyone, so that the risk of testing in production can be reduced. The idea is to get feedback, fast.. if something goes wrong, reverting is automatic and without any hassle. We believe that taking steps towards these directions is both natural and necessary in order to stay true to our customers and deliver a compelling product for Continuous Delivery.
The Release stage is made up of several categories:
Deliver your changes to production with zero-touch software delivery; focus on building great software and allow GitLab CD to bring your release through your path to production for you. This category is at the "complete" level of maturity.
Management and orchestration of releases-as-code that supports intelligent notifications, scheduling of delivery and shared resources, blackout periods, relationships, parallelization, and sequencing, as well as support for integrating manual processes and interventions. This category is at the "minimal" level of maturity.
Use any static site generator to create websites that are easily managed and deployed by GitLab. This category is at the "complete" level of maturity.
Get a full production like environment for every merge request that updates on each commit. See code running and enable user acceptance testing before you merge. This category is at the "complete" level of maturity.
Mitigate the risk of production deploys by deploying new production code to small subset of your fleet and then incrementally adding more. This category is at the "minimal" level of maturity.
Feature flags enable teams to achieve CD by letting them deploy dark features to production as smaller batches for controlled testing, separating feature delivery from customer launch, and removing risk from delivery. This category is at the "viable" level of maturity.
Release Governance includes features such as deploy-time security controls to ensure only trusted container images are deployed on Kubernetes Engine, and more broadly includes all the assurances and evidence collection that are necessary for you to trust the changes you're delivering. This category is planned, but not yet available.
Use Vault embedded in GitLab for managing your secrets both inside and outside of GitLab. This category is planned, but not yet available.
It's important to call out that the below plan can change any moment and should not be taken as a hard commitment, though we do try to keep things generally stable. In general, we follow the same prioritization guidelines as the product team at large. Issues will tend to flow from having no milestone, to being added to the backlog, to being added to this page and/or a specific milestone for delivery.
needs: to specify a job that can always be started immediately, regardless of stage FY20 Vision
.gitlab-ci.yml, auto-configure GKE FY20 Vision
needs:to refer to a job in the same stage FY20 Vision
allow_failureaction for pipeline rules FY20 Vision
new-branchkeywords for only/except