Gitlab hero border pattern left svg Gitlab hero border pattern right svg

Product Section Vision - CI/CD

CI/CD Overview

The CI/CD section focuses on the code build/verification (Verify), packaging/distribution (Package), and delivery (Release) stages of the DevOps Lifecycle. Each of these areas has their own page with upcoming features, strategic considerations, and more; this page ties them together via the important concepts that unify the direction across CI/CD. Also related is the mobile use case, which touches on features across GitLab but has a large component in making build, test, and release for mobile apps better and easier to do. You can see how the CI/CD stages relate to each other in the following infographic:

Pipeline Infographic

CI/CD represents a large portion of the DevOps market with significant growth upside. The DevOps Ready portion of the market, representing application release automation/CD and configuration management (mainly aligned to the Release stage) supported 984.7MM USD revenue, projected by 2022 to grow to 1617.0MM. The DevOps Enabled portion of the market, representing Continuous Integration, integrated quality/code review/static analysis/test automation, and environment management (mainly aligned to the Verify stage) represented an even larger 1785.0MM market, projected to grow to 2624.0MM. With a combined projected total of 4241.0MM by 2022, it's critical that we continue to expand our share of this important segment. Data Link, GitLab Internal

The CI/CD section currently has 41 members with 22 vacancies open. Tracking to our hiring plan can be seen on our hiring chart. Verify is in year 7 maturity, Package is year 3 maturity, and Release is year 4 maturity.

Competitive Space and Positioning

CI/CD is a highly competitive and evolving area within DevOps, with a rapid pace of technological innovation. Kubernetes itself, a massive driver of innovation, just turned five years old in 2019. Docker, now considered a mature and foundational technology, was only released in 2013. We do not expect this pace of innovation to slow down, so it's important we balance supporting the current technology winners (Kubernetes, AWS, Azure, and Google Cloud) with best-of-breed, built-in solutions while at the same time avoiding over-investment on any single technology solution such that could make us irrelevant as it becomes dated.

There are dangers on both sides; Pivotal resisted embracing Kubernetes early and has suffered for it. At the same time, technologies like Jenkins that too fully embraced what are now considered legacy paradigms are enjoying the long tail of relevancy, but are having to scramble to remain relevant for the future resulting in confusing product messaging and adoption challenges.

Trend Towards Consolidation

Since the beginning of 2019, we’ve seen a trend towards consolidation in the DevOps industry. In January, Travis CI was acquired by Idera, and in February we saw Shippable acquired by JFrog. Atlassian and GitHub now both bundle CI/CD with SCM, alongside their ever-growing related suite of products.

It's normal for technology markets go through stages as they mature: when a young technology is first becoming popular, there tends to be an explosion of tools to support it. New technologies have a lot of rough edges that make them difficult to use and early tools tend to center around making the experience easier to adopt and use. Once a technology matures, tool consolidation is a natural part of the life cycle. We're in a great position to be ahead of the curve on consolidation, but it's a position we more and more are going to need to defend against well-funded competitors who also have a vision for a single DevOps application.

Making Continuous Delivery Real

Our product positioning for CI/CD is aligned to the one set forth in Jez Humble's classic Continuous Delivery book. The ideas of reducing risk, getting features and fixes out quickly, and reducing costs/creating happier teams by removing the barriers to getting things done have stood the test of time. We've also embraced recent innovation happening here that ties these original concepts into Kubernetes and incremental rollout, that is emerging under the banner of Progressive Delivery. Still, the 5 principles at the heart of continuous delivery remain illustrative of the problems we are focused on solving in our CI/CD solution:

  1. Build quality in
  2. Work in small batches
  3. Computers perform repetitive tasks, people solve problems
  4. Relentlessly pursue continuous improvement
  5. Everyone is responsible

With a head start on a single DevOps application, we're uniquely positioned to make Continuous (and Progressive) Delivery a reality for our users, supporting them through the transformations they need to achieve these goals. No point solution is able to provide a comprehensive view on software delivery in the way that we are, unifying execution teams rather than providing segmented tools/views for each department or role.

Challenges

Opportunities

Strategy

Our unified strategy for CI/CD is oriented around the idea of enabling GitOps enabled workflows throughout our single application. GitOps is an emerging concept that wraps all of CI/CD into a git-first workflow: pipeline configuration, code, and infrastructure are all stored in git as the single source of truth, and the entire development flow from ideation through delivery is interacted with in git-first tooling. Events in the system are oriented around state changes relating to source control, issues and merge requests, facilitating change management that is easier to use than previous paradigms, and all of this is tightly connected through a single GitLab pane of glass. All of this is to enable the continued evolution from waterfall (week or monthly cycles), to agile (cycles in terms of days), to DevOps (real, immediate feedback through progressive delivery.)

GitLab is uniquely positioned to make this reality, being a provider of a mature SCM and CI/CD solution that has always been git-first. Disparate sources of truth for the state of the world across a fleet of point applications just doesn't scale, and we believe this is a better way. CI pipelines flowing information about builds into the package management system and ultimately out via progressively-deployed releases, all with no setup required, and all using git as the common reference point will be very powerful. A great example of the kind of features enabled by this connectedness is reflected in our vision for automatic evidence collection in the release process.

We also have a specific strategy for each stage, focused on the unique elements of that area:

Verify

Users of Verify's CI solution can expect us to invest heavily in multi-platform support, making it just as easy to build on Windows and MacOS as it is to build on Linux today. In terms of pipeline capabilities, we've recently implemented directed acyclic graphs, a powerful primitive for complex dependency relationships, and we will expand on this with great features like child/parent pipelines and cartesian products for jobs. The innovation never really stops here: there's always a way to provide a new, powerful command that can help organize your pipeline configuration in a simpler and more expressive way.

Beyond pipeline execution, we'll improve how we present and act on interesting data around what jobs do and how they perform; timing and efficiency information is one part, but so is how test results are changing over time. Our epic for CI/CD analytics captures some of the important metrics we've identified so far. This data then can be acted upon by GitLab itself to automatically tune your pipelines for better performance - imagine automatically tuning parallelization of your test runs - without users having to spend time thinking about it. Self-healing pipelines that automatically adapt to changes to maintain their efficiency will be a powerful tool for teams.

Package

The goal of the Package group is to build a set of features that within three years, will allow ninety percent of our customers to store all of their packages and artifacts in GitLab. In order to accomplish this goal, we will expand the footprint of our existing product, build products that fully leverage the benefits of GitLab's single application, and unify the container registry, build artifacts and the package registry to create a seamless, unified user experience for building, testing and releasing code.

We will first focus on improving the usability of our existing registries. By providing a consistent and delightful user experience across integrations, we will be able to quickly introduce new package manager integrations. Next, we will ensure that all Package features take advantage of GitLab's single application. We will simplify the lives of our users by standardizing authentication and permissions for each of our package manager integrations. We will help our users by creating visibility and transparency into their registries by providing meta data and evidence for how images/packages are built, by whom and when.

Then we will work to improve the performance and reliability of builds by rolling out the dependency proxy to each package manager. Increased adoption of the dependency proxy will allow us to build new features that will help our customers to reduce open source risk by providing visibility and transparency into external dependencies.

Finally, we will increase adoption and conversational development by providing improved discovery and collaboration features, such as search.

Release

The traditional roles of software development teams are changing. The trend in the market is to build self-managed teams and we believe that more and more teams will adopt this change in the next three years. Gitlab sees the importance of giving these teams everything they need in order to perform their job without additional overhead. We have seen the traditional centralized Release Management role at companies change, becoming more of a shared responsibility that is distributed amongst several groups, and expect that to continue.

To support this transition we are bringing Release Management features into the product, such as release orchestration, runbooks for releases, management of blackout periods, environment scheduling, etc., but in a way where these features are integrated directly into the single application. Instead of having a Release Manager view a dashboard within the product, feature flag states are connected with the issues and merge requests that introduced them. Releases automatically collect evidence about what the state of issues, team members, test results, and so on were at the time of the automatically, and these are presented simply within the release view. This way of working serves both the teams writing the software and the teams delivering it, whether they are separated or not, and is a unique solution in the market compared to other point tools which provide solutions to Release Managers but do not really integrate well with the rest of the team.

In addition to the change in roles, the next natural step of SW delivery is progressive delivery. Once there is an automated CI/CD process in place, the next part of the journey is progressive delivery. Progressive delivery controls the deployment to a specific subset of users before being fully shipped, so that testing in production doesn’t have to be a massive risk. The idea is to get feedback, and fast. If something goes wrong, reverting is automatic and without any hassle. We believe that taking steps towards these directions is both natural and necessary in order to stay true to our customers and remain a unified tool for the entire DevOps lifecycle.

Plan & Themes

Given our vision and strategy, our plan for the next year has been organized around a number of themes that support our direction. These are listed here in roughly priority order:

What we're doing

Single Application CI/CD

DevOps tooling is heading towards consolidation simply because users prefer well-integrated solutions that don't require administration effort to keep everything wired together in order to realize the benefits of having all the information you need in one place. Point application solutions like TravisCI and CircleCI are struggling under this paradigm because they suffer from the fundamental problem that CI/CD is not an island. In order to make sure their solutions can compete effectively they are forced to maintain integrations with various stacks, which is expensive in terms of focus and results that they are unable to innovate as fast as competitors that are providing a unified solution.

Ultimately, our single application solution is what will help us ensure that GitLab is a Robot Cat instead of a Sheepdog. By embedding CI/CD naturally into everyone's workflows, as a first-class part of working with issues, source control, merge requests, etc., CI/CD isn't a box under someone's desk that nobody knows how to work while on vacation; it's a part of how everyone works together - the glue that binds everyone around how they deliver software together as a team.

Related Epics:

Multi-Platform Support

GitLab has traditionally done very well with building and deploying software on Linux, with our shared runner fleet making it very easy to get up and running quickly. We've supported MacOS and Windows Runners, but only in a bring-your-own configuration where you host and register your own Runners. Our top priority is changing that by improving Runner features for these platforms in addition to providing them as available options in the shared runner fleet.

Just being able to run builds on the platform isn't sufficient, though. We also want to improve our support for unit testing and code coverage frameworks used on these operating systems, as well as our support for different package managers that are used by code build there.

Related Epics:

Speedy, Reliable Pipelines

At the core, GitLab CI/CD was built from the ground up with speed and scalability in mind. This is reflected both in the product design and product vision (automated testing, testing all the things) To enable our users to go from monthly shipping releases to truly enabling continuous delivery, everything that we do must be concerned with the scalability and speed of continuous integration and testing. Not that sword fighting on office chairs isn't fun, but we do want people to be able to contribute at the speed of development.

Time to First Byte is the first measure we're improving, which is the time it takes to set up the execution agent for every job, before the first line of your script is run. This can be fetching the image, provisioning a host, running internal setup, polling from the runner, and so on. All of these add delays to every job, but do not contribute to actually doing the important thing that needs to be done. To track and understand how we measure this, see the time to first byte child epic. This also includes elements of improving the way the fleet of shared runners works, such as re-enabling registry mirroring ([infrastructure#3158(https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/3158)), and doing things like pre-creating the runner environment for upcoming jobs

Wall clock time for a pipeline is also an interesting measure. This is the one you really feel - when you make a commit and a pipeline starts, and you need to go have lunch before you see the result, you're really suffering. Our vision here is to have GitLab help you identify the bottlenecks in your pipelines and ideally remediate them automatically, but at least make it visibile where the bottlenecks are so you can investigate. Issues like our auto-determining the test parallelization factor represent how we want GitLab CI/CD to automatically find and solve these problems for you, and our dependency proxy is part of how we're building speed into the workflow itself.

Related Epics:

Do Powerful Things Easily

One of the core principles of GitLab CI/CD is the idea that we want to give people powerful tools to do their job; flexibility is key, and we want to ensure that getting complex things done with small, easy-to-use, but powerful primitives is the GitLab way. This also helps us remain flexible and avoid hewing too closely to a specific technology solution, ensuring we continue to diverse and sometimes legacy technology stacks, and retain our flexibility to embrace future innovation. Put simply, we will always prioritize powerful building blocks in GitLab that are exposed to you, the user. Where we have structures to get started quickly with a given stack, those will be built up from the same primitives that you have access to as well.

An interesting measure for this is how long it takes to get to your first green build from the time you create your project, for both new and seasoned pipeline users. We are going to start tracking this as a success metric for how easy it is to get in and get your work done; GitLab's built-in functionality should make this quick and easy.

Related Epics:

Progressive Delivery

Put simply, Progressive Delivery is a set of emerging CD best practices, oriented around being able to control and monitor deployments incrementally over time, and in an automated and safe way.

Progressive delivery controls the deployment to a specific subset of users before being fully shipped, so that testing in production doesn’t have to be a massive risk. The idea is to get feedback, and fast. Similarly to Compliance as Code, our single application is uniquely capable of providing an integrated solution in this space by using feature flags, incremental rollout and other traffic routing methods that can be controlled directly on the deployment pipeline. You can control which users and what features can be accessed in addition to where and when, giving you ultimate control of your production environments and given you the ability to make quick decisions when it matters.

Additional articles on Progressive Delivery can be found on the LaunchDarkly blog, RedMonk and The New Stack, as well as our own blog posts highlighting Feature Flags and Review Apps.

Delivering software to your environments automatically is the first step, and modern technologies like Kubernetes are now allowing on-demand environments to be created like never before. By tying your source repo to intelligent infrastructure that understands the intentions of developers, you can enable a true one-click workflow to deliver to your internal or external end users.

Related Epics:

Mobile Support

Mobile is an important segment of the market - many consumer brands, even those that would be considered non-tech are now development and distributing mobile apps as part of their customer engagement strategies. It's important for GitLab that we support these customers to be successful in this area. To see an example of where we're headed and the kinds of easy-to-use delivery flows we have in mind, we have a video where we demonstrate (using GitLab today) making a change to a mobile app using an iPad Pro, sending the build over to a MacOS runner, and receive a working copy back to test on the very same device using TestFlight. Part of supporting Mobile builds also includes our vision for better multi-platform support, making shared runners available for MacOS in particular will allow for easier iOS builds on GitLab.

More information on how we plan to support mobile CI/CD can be found on our mobile use case page.

Related Epics:

Compliance as Code & Secure Secrets

Compliance as Code is the idea that requirements for auditing and compliance teams can be collected with zero additional cognitive load from developers and other team members participating in software delivery. Because GitLab has an end-to-end view of the artifacts that are associated with releases (issues, merge requests, feature flags, and so on) we are uniquely positioned to provide a comprehensive view of compliance-related activity, collected on an ongoing basis throughout the build, test, and release process. Our vision here begins by improving our releases feature to start collecting evidence from the cycle leading up to the release event (gitlab-ce#56030).

In addition to collecting compliance evidence along the path to production, our plan is to improve the way GitLab pipelines interact with secure secrets, typically held in a Vault or other secrets store. Our plan is to first allow GitLab to integrate with your existing Vault, implemented via gitlab-ce#61053. In the future, however, we have a broader vision for bundling Vault with GitLab and making secrets management a part of the GitLab product in general; this includes storing GitLab's own secrets in an external Vault.

Related Epics:

Universal Package Management

Developers rely on a variety of language-specific package manager clients to access, build and publish packages. Our vision is to provide a universal package manager, one that integrates with the most commonly used clients and makes them accessible in a single, shared location. Success is defined by our ability to not only support a wide variety of package manager clients, but to provide a consistent, delightful user experience with each integration. We currently support Maven, NPM and Docker. We will be adding support for C++ (Conan), .net (NuGet), Linux (Debian, RPM), and Helm charts (Kubernetes).

Most universal package managers in the market today work as point solutions that integrate with other services for source code management, continuous integration and deployment. This introduces risk for administrators who are forced to support multiple platforms and multiple authentication protocols. And it introduces inefficiencies for developers who face unnecassary context switching and increased complexity.

We believe that by building a universal package management system into GitLab, that we can help solve those problems on behalf of our users. It is critical that we continue to build a solution that leverages the benefits of GitLab's single application. Beyond authentication and colocation, we will ensure that we utilize GitLab's data to help provide context, evidence and an optimized user experience as you build, test and deploy your code.

Related Epics:

What we're not doing

Event-Based CI/CD

We've seen GitHub actions v2 come out with a more event-based model for building up CI/CD pipelines. Distributed, event-based systems in general can be very powerful, but as complexity scales they tend to add to the complexity rather than help you solve it. We believe that to a large extent, building and maintaining a CI/CD platform should be and exercise in minimizing complexity rather than embracing it. There's nothing that says CI/CD has to be hard, and we believe our vision for powerful, small primitives will scale better and result in happier, more productive teams in the long term.

That said, in certain simple cases (and especially when it comes to getting up to speed quickly) things like webhooks can be a powerful tool. We don't believe these should be avoided at all costs, we just don't believe that these are a stable foundation to build a delivery platform on top of and have no plans to build an event-based model for GitLab CI/CD in the next twelve months.

GitOps Pull-Based Deployment

One proposed aspect of GitOps, apart from keeping Git as the source of truth and doing operations by Merge Request which we have embraced, is inverting the deployment model to be from push (where jobs execute and push changes to an environment, which can then be monitored for divergence) to pull (where the environment itself runs an operator that monitors Git and itself, ensuring that everything stays in sync.) There are potential control benefits here, in that the CI/CD system does not need to maintain access to the target environment, but the reality is is that the CI/CD system will continue to be able to influence the environment in various ways by changing state in Git. Additionally, bringing the state change automation inside the cluster brings benefits to observability by seeing the environment and the state change in a single place.

For us, the benefits here are less clear than the core GitOps principles of everything-as-code, everything in Git, and change via Merge Request. We'll continue to monitor how this area evolves, and potentially change course as more innovation or clear benefits to this aspect are realized, but in the meantime we are not investing in this area.

What's Next

With the above vision, strategy, and plan accounted for, here's our plan for what we will be delivering over the next few releases:

12.3 (2019-09-22)

Verify

12.4 (2019-10-22)

Verify

Package

Release

12.5 (2019-11-22)

Verify

Package

Release

12.6 (2019-12-22)

Verify

Package

Release

12.7 (2020-01-22)

Verify

Package

Release

12.8 (2020-02-22)

13.0 (2020-04-22)