Everyone can contribute! Feel free to comment on our async AMA issue, email Jackie Porter, and propose an MR to this page!
The Verify Stage is responsible for executing on the market needs for continuous integration. From our Continuous Integration use case:
When practicing Continuous Integration, teams collaborate on projects by using a shared repository to store, modify and track frequent changes to their codebase. Developers check in, or integrate, their code into the repository multiple times a day and rely on automated tests to run in the background. These automated tests verify the changes by checking for potential bugs and security vulnerabilities, as well as performance and code quality degradations. Running tests as early in the software development lifecycle as possible is advantageous in order to detect problems before they intensify.
The total addressable market (TAMkt) for DevOps tools delivering against the Verify stage is $1.7B in 2020 and is expected to grow to $2.80B by 2024 (15.09% CAGR) (i). The Verify Stage makes up 23% of the Ops market and represents a significant portion of GitLab's expanding addressable market.
The Verify Stage has continued to maintain a premium experience for individual and small teams of Software and DevOps Engineers with market share increasing each month as evidenced in our Verify product performance indicators (internal).
Delivering on the Enterprise use case is steadily increasing as evidenced in our Verify Paid user-product performance indicators (internal). To continue this growth, the Verify Stage needs to invest more in the scaling requirements for large organizations, deliver on a solution for building secure and compliant software, as well as prioritize the usability of our core CI capabilities.
Combining GitLab's Biggest Risks with the Verify Stage perspective, the Continuous Integration vision has some significant strengths, weaknesses, opportunties, and threats to becoming the leading platform for building, testing, and optimizing code:
| STRENGTHS Internal resources to exploit |
WEAKNESSES Internal Risks to mitigate |
OPPORTUNITIES External resources to exploit |
THREATS External Risks to mitigate |
|---|---|---|---|
| We are one of the core adoption paths for our users at GitLab Developer first approach for experiences Meaningful insights from use of the DevOps platform |
Lack of usage data-informed product decisions Ineffectively managed Technical debt/bugs Over indexing to Enterprise Product Management |
Reduce friction between all functions of development in a single-platform Empower developers to manage operations, quality, and security by baking those activities into GitLab |
Competition is by far the largest external risk for GitLab GitHub Circle CI JFrog HashiCorp Public cloud providers New market entrants |
As organizations migrate to a cloud-first strategy, Continuous Integration must work to adapt to any changing needs in scale, performance, and usability. The Verify Stage must simultaneously support the trend toward microservices architecture and infrastructure as code, while balancing the needs of monorepos. In response to the rise in supply chain attacks, there is an ever-increasing pace of government-issued directives, standards, and regulations focused on the security and integrity of the software supply chain. This means we must add features and capabilities that enable customers to efficiently meet the most stringent secure CI/CD and software chains of custody requirements. In order to adequately deliver on these expectations, the core competencies Continuous Integration must meet for Enterpises are as follows: build and test automation, pipeline configuration management, visibility into CI performance, and built-in compliance with security.
Our top competitors for the Verify Stage are as follows:
Secondarily, there are emerging competitors in the continuous integration space we are watching carefully:
Our mission for GitLab Continuous Integration is to empower all users to easily contribute to the automated building, testing, and optimization of code across teams, organizations, and the Open Source Community.
From this mission, the GitLab Continuous Integration three-year vision is to become the leading platform for Software Engineers, DevOps Engineers, and Development Team leads for automatically building, testing, as well as optimizing code.
We will execute against this vision by:
.gitlab-ci.yml files.In FY23, we are expecting on moving the vision forward by delivering on our Product Themes:
Each of our product groups have specific JTBDs:
Some of the core JTBDs for our three year vision and strategies are as follows:
Enabling the Continuous Integration use case is currently spread across four product groups (Pipeline Authoring, Pipeline Execution, Pipeline Insights, and Runner). In order to define our top focuses and intiatives, the Verify Stage needs to have a concerted perspective on what GitLab will offer for Continuous Integration. We are targeting the Ops Section Direction personas and will support these users with our Continuous Integration mission, vision, and one-year direction.
The Continuous Integration vision is a longer term investment. A core capability GitLab CI unlocks is the DevSecOps workflow. A key focus for the Verify Stage is supporting the automation of secure workflows in GitLab. In the next year, we expect to make progress on the following categories to support our target of being the leading CI platform of choice:
.gitlab-ci.yml files.There are important things we won't work on to focus on our one year plan.
In the Verify Stage, we have several investment cases highlighted as top priorities for future investment in the Ops Section. Additionally, there are investment cases beyond the ones identified stack ranked by potential return on investment as indicated by priority below:
Gain the confidence to ship at blistering speed and immense scale with automated builds, testing, and out-of-the-box security to verify each commit moves you forward. This category is at the "complete" level of maturity.
Learn more • Documentation • Direction
Watch us reach 20M builds per day on GitLab.com.
Category containing features associated with editing/authoring your .gitlab-ci.yml.
Learn more • Documentation • Direction
Manage secrets and protect sensitive data to enable GitLab, or a component built within GitLab to connect to other systems. This category is at the "minimal" level of maturity.
GitLab Runner is the execution agent that is used to run your CI jobs and send the results back to GitLab.
GitLab hosted Runners for GitLab CI on SaaS
GitLab Runner fleet management at enterprise scale
The Jenkins Importer helps unblock teams from migrating to GitLab CI. This category is at the "viable" level of maturity.
Code testing and coverage ensure that individual components built within a pipeline perform as expected, and are an important part of a Continuous Integration framework. This category is at the "viable" level of maturity.
Keep build artifacts under control and actionable.
Be confident in the performance of your changes by ensuring that they are validated against real world scenarios. This category is at the "minimal" level of maturity.
Keeping master green and ensuring the stability of collaboration on branches is vitally important. GitLab has introduced Merge Trains as an important way to accomplish this. This category is at the "viable" level of maturity.
Get a fully functional pre-production environment for every merge request that updates on each commit. See code running, and enable user acceptance testing and automated smoke tests before you merge. This category is at the "complete" level of maturity.
Priority: low • Learn more • Documentation • Direction
The Verify group is at the entrypoint to the funnel in our user adoption journey, so our features are an critical enabler for users seeking a complete DevOps platform. While we do try to drive adoption in order to support multi-stage growth at least partially through features at the free tier, it's also important for us that we have features at the paid tiers. For our group, these will typically be cross-department and cross-company views related to CI templates, quality and pipelines. See below for the how we are thinking about each of the tiers, and the kinds of features that will be included.
The foundation of the Free strategy for Verify is that enhancements to baseline CI YAML features will be available in this tier by default. The rationale for this approach is that we want to preserve a consistent experience. Users must always be able to use the same .gitlab-ci.yml in all tiers.
Beyond this, features that drive broad adoption and help with onboarding (including generally making it easier to get started with GitLab CI) are also good candidates for inclusion in this tier. Providing solutions to simplify the deployment and management of Runners at scale for self-managed is also critical for all tiers of users.
The default paid tier for enterprises, Premium will cater to directors operating a medium to large instance. We will focus on features that solve for typical entry-level enterprise needs: reporting and analytics supporting Ops Insights, operational efficiency driving effective project management, and supporting visibility in consumption related to 10,000 CI/CD minutes per month medium to large organizations.
You can see features that we are considering including in this tier in this issue search.
Using GitLab CI across hundreds or thousands of projects in large enterprises means a greater need for portfolio management and consistency of experience in development practices. This is accomplished by making templates discoverable via gitlab#320698 and a consistent authoring experience such as in these issues. Additionally, the larger the organization the greater the need for regulation and compliance which is where features like Protected Runners captured in gitlab&6058 or better integrations with Compliance Pipelines become especially attractive. For customers who self-manage a Runner Fleet, ensuring the security and compliance of the Runner execution environment is critical. Features such as the auto-removal of inactive runners and providing reporting and automation to manage outdated runner versions are just a few examples of features coming in Ultimate aimed at simplifying Runner operations and maintenance. To secure the flows for HashiCorp Vault users, we would like to automatically mask any Vault secrets that are fetched by GitLab CI. To ensure better pipeline compliance we plan to provide you with the ability to enforce the presence of specific variables when validating pipeline configuration. Lastly, our core promise for the Ultimate tier is enabling users to effectively monitor and best use their 50,000 CI/CD minutes by setting CI/CD Minute Limits on their projects.
You can see features that we are considering including in this tier in this issue search.
As a part of our annual planning, we want to transparently convey our progress to our goals and what lies ahead for each of the groups in the Verify Stage. We use a format called "Letter from the Editor", a play on the Letter to the Editor, which positions the writer of this letter to the community of GitLab rather than the community writing to GitLab. We use this method to broadcast our priorities and updates in the product group.
To the GitLab Community and customers,
First I want to thank our community for all the community contributions over the 15.1, 15.2 and 15.3 releases. Thank you!
At the beginning of FY-22 the team was focused on defending our position as a market leader in Continuous Integration. We looked to solve the problems customers had with long-running pipelines or pipelines that failed unexpectedly by collecting and presenting data in the GitLab UI. Over the course of the year, we shifted focus to ensure the reliability of Continuous Integration on gitlab.com which took the majority of our time and was the right call at the time. We also re-evaluated our maturity level and found some rough edges in the capabilities of the Continuous Integration category so moved it from Loveable to Complete.
Throughought the first half of the Fiscal year we have been focused ensuring OSS contributors can continue to use the GitLab DevOps platform while mitigating crypto mining abuse. We have made improvements to visibility of CI/CD Minutes duration tracking before we started counting minutes for all public projects on June 1. Our focus now turns to ensuring contributors to Open Source Program projects and GitLab can run pipelines for contributions without having to pay to do so.
Our PM and Product designer are currently exploring solutions to letting project owners allow trusted external parties run pipelines in their projects to solve for use cases like keeping dependencies up to date without manual efforts. Beyond that, we are looking at a broader theme of extending our lead in CI/CD by helping solve problems users are having with debugging pipelines and continuing to move forward with the vision for merge trains.
Looking forward if we wanted to accelerate the maturity plans of our categories we may look for more ways to include the wider community for contributions to the core CI functionality. Features such as Running GitLab Pipelines for external SCM, Merge Trains, and pipeline views are key areas to enhance. Activating the community will avoid decreased velocity in feature development while being able to maintain focus on stability and performance of the GitLab platform.
An additional thought experiment is to increase investment in Pipeline Execution by 2-3 backend engineers. This increase would enable us to effectively burn down our bug backlog and accelerate forward-looking technical roadmap items related to the Continuous Integration Scaling category.
First and foremost, I would like to thank our community of users and the Pipeline Authoring team for an amazing FY22. The team was able to deliver a significant amount of value while maneuvering between different priorities, directions, and requirements. The community of users helped us shape the product and solution by actively engaging on issues, threads, and comments. Thank you for all the hard work and dedication you put into the product. In Q1 the pipeline authoring had introduced the Pipeline Editor, throughout this year, we've improved the Editor with additional capabilities. In addition, our team is responsible for the CI syntax and the pipeline graph, which means there is a constant demand to support, fix and enhance our capabilities around authoring and visualizing pipelines. Later this year, the team was handed over the responsibilities for CI variables, with a rich feature set and a growing number of issues (approximately 20% of our backlog). In the coming FY23 the team needs to continue its focus on growth by focusing on the Pipeline Editor, UI improvements, CI templates, and variables, however, without increasing our current headcount, it would be hard to achieve.
Looking back on the list of accomplishments in FY22 our year had started with the release of the Pipeline Editor with the purpose to improve the way users author their pipeline. The Editor comes with key capabilities around linting, validation, and visualization of a pipeline (before doing a single commit). The Editor comes with a branch selector that allows users to leverage the Editor and its capabilities while working on a non-default branch. The pipeline Editor received positive feedback by our users and today has approximately 16K unique users per week. In addition we've refactor and improved the pipeline graph performance and introduce the Dependencies view to better support DAG users. We succesfully Matured Pipeline Authoring category from minimal to viable, and deliver some of the most popular issues from our backlog, here are just a few examples:
Currently, Pipeline Authoring is composed of 5 Engineers (2FE + 3BE), 2 EMs, 1 SET, and 1 UX designer. We currently have an opening for a frontend engineering if you are interested in applying! With this backfill of a FE engineer, the team is the exact size we started a year ago. The increasing maintenance cost for the pipeline editor and the addition of CI variables, I have concerns about our ability to continue to focus on growth in FY23.
Over the last year, we saw how our users adopted our features. I believe that our team needs to continue its focus on growth, our solution is still young with many opportunities to invest in, major themes we'll try to focus on in the coming year are. Double down on the pipeline editor - The Editor is one of our differentiators in CI. It is our team's growth engine today. We've seen evidence of the growing number of users leveraging it and its capabilities. We should continue our investment in the Editor. Some of our plans for FY23 is adding Auto-complete, Pipeline simulation,Add include: file links to the pipeline editor and improving the user experience. Those improvements are also likely to affect our SuS score significantly.
GitLab CI has many capabilities; however, in some cases, our UI for those capabilities is still in the MVC phase. The UI experiences often are non-intuitive or missing additional functionality. Our users expect to complete those missing capabilities and improve their user experience.
In addition to usability in our MVCs, environment variables are table stakes for CI solutions. With many popular issues, variables are now 20% of the team's backlog and could drive PGMAU for Pipeline Authoring team. Failing to invest in this key feature would lead to customer frustration, churn, and loss of revenue, more information on the variables investment case.
As outlined in the following Opportunity canvas (pending approval), the CI catalog is one of the most popular ask/requirements that come out from many customer conversations. A CI catalog could be a growth engine for GitLab, It can encourage cross-stage collaboration, potentially drive high impact to our Stages per User (SpU) metric. A CI catalog is also highly coupled and compliments our CI/CD templates vision, which serves as the main JTBD for large enterprise users.
As mentioned above, the team size has not been changed. However, more responsibilities were added in FY22. Looking into FY23, adding 3 engineers as outlined in this business case would empower our group to deliver a complete variable solution and clear the path for the rest of the team to focus on the rest of our initiatives. Without this, our team would have to drop the work for templates and provide a partial solution for variables.
To the GitLab community, customers,
First, thank you for your ongoing support, contributions, and, yes, patience. Your engagement enables us to continue to expand the GitLab Runner supported platforms and compute architectures, which helps deliver our product vision of GitLab as the single DevOps platform.
As an introduction, I am Darren Eastman, the Runner Product Manager. We have had new exceptional individuals join the team; Gina Doyle, UX designer; Romuald Atchadé, back-end engineer; and Miguel Rincon, front-end engineer. Steve Azzopardi, a mainstay of the team and architect of many critical features, has transitioned to another team at GitLab. I would like again to thank Steve Azzopardi for his exceptional contributions and the rest of the Runner team, who have delivered excellent work this past year. There are also two open backend engineering positions in Runner, so if you or someone you know would be interested, please apply!
As we head into GitLab Fiscal Year 2023 (February 2022 to January 2023), we have many value-added features and capabilities that we hope to deliver across the GitLab Runner categories (Core, SaaS, Fleet.) For SaaS, that includes introducing a new autoscaler for the Linux Runners with Docker Executor and transitioning the Windows and macOS Runners offerings to general availability. In Runner Core, we will balance investing in reducing the backlog of aged bugs and technical debt items with investing in new features. Finally, for Runner Fleet, we plan to accelerate the pace of innovation in bringing to market new features and capabilities to cost-effectively manage Runners at scale so that you can efficiently and reliably execute CI/CD workloads for your organization. That effort is currently decomposed across the following epics; Runner Fleet: FY22 to FY23 Q2, Runner Fleet: FY23Q3 - FY23Q4
In January 2021 (FY22), I recorded a video describing the Runner's vision and direction. The slides also included a view of the potential evolution of the market landscape for computing operating systems and architectures. Linux, macOS, and Windows were highlighted on that slide as anyone will, of course, expect those operating systems to continue to be widely used over the next five to ten years. The highlighted compute architectures were x86-64, AArch64, Apple MI. Since then, we have added support to the Runner for the IBM Power platform, ppc64le architecture. We are also planning on adding native support for z/OS on the IBM Z platform, s390X compute architecture. In short, the operating system and compute architecture landscape that we are using to manage our investments in Runner Core has expanded, and quicker than I had anticipated at the start of this year. Beyond the investments in supporting GitLab Runner on the major compute architectures, there were other critical features and capabilities that I had planned as themes for the roadmap. Well, what have we done so far? For the GitLab Runner autoscaling on Kubernetes theme, we did, in fact launch the Runner Operator for Red Hat OpenShift. In addition, we were able to maintain a continued level of investment in the Kubernetes Executor. In looking at the numbers, this translates to approximately 30% of issues delivered for Runner Core. And these issues that we shipped for Kubernetes are proving to be extremely valuable to our community and customers. And there is more to come in the 14.6 release. While the Kubernetes executor story is good, we did not progress on a few other critical themes for Runner Core. First, we planned to start re-architecting the Runner API security mechanism. This is critical to support the continued adoption of GitLab CI, especially for organizations with stringent security and compliance requirements. Secondly, we did not make any meaningful progress on the Docker Machine autoscaling replacement for customers that self-manage Runners on virtual machines hosted on GCP, Azure, AWS and rely on the simplicity and effectiveness of this solution. We delayed work on these two efforts as we had to reallocate engineers to support work on GitLab SaaS scalability. The impact of the reallocation to support GitLab SaaS scalability also resulted in a stoppage of work on the necessary items required to transition the Windows and macOS SaaS Runners to GA. On the Runner Fleet front, thanks to the efforts of the small but dedicated team, we have started to make progress on foundational features. However, we certainly had a goal to be much further along in the evolution of this solution.
We have ambitious goals for the Runner Group in FY23. The complexity of the work that's planned cannot be understated. In Runner Core, we need to start re-architecting the API security mechanism. We also need to maintain our Docker Machine fork for autoscaling on public cloud VM's. In parallel, we will likely have to develop a new autoscaling solution. This is even though we offer autoscaling Runners via the Kubernetes Executor. However, customer feedback indicates that a VM-based autoscaling solution on public cloud providers is still highly desired. We plan to transition the Windows and macOS SaaS Runners to GA. In parallel, we will be working on re-architecting the Docker Machine autoscaler that currently powers the Linux SaaS Runners. This brings back the old metaphor of changing the tires while the truck is speeding down the highway at seventy miles per hour. Finally, we want to make meaningful progress on providing value-added features that simplify the management of a fleet of Runners.
While we have a fantastic team, we will need to increase the team size to keep pace with the demand of maintaining and improving the core runner product, not to mention delivering on the new features and capabilities for the Runner SaaS and Runner Fleet categories. In the Verify: Support Runners as a service - Runner SaaS investment case we have asked for 8 headcount to deliver a new autoscaler for the Linux SaaS Runners, Windows Runners GA, and macOS Runners GA). Without this additional investment, our delivery velocity will be extremely slow. This will translate into critical initiatives taking months to complete with delivery well into FY24.
Hopefully, this provides you with deeper insights into our thought process and challenges as we head into next year.
From the Runner Group - Have a great holiday season, Darren Eastman.
To GitLab Community and Pipeline Insights Users,
FY23 is bringing a lot of opportunity for the Pipeline Insights Team. This letter will walk you through the team changes, what we thought we would accomplish in FY22 and what we are expecting to accomplish in FY23. We also have a thought experiment where we accelerate our direction by suggesting an increase in headcount.
First things first, my name is Jackie Porter and I am the Acting Product Manager for the Pipeline Insights Group. We have also had some changes in the Pipeline Insights team over the past 12 months. I'd like to introduce our Fullstack Engineering Manager, Scott Hampton, who has been our fearless leader over the 4 Pipeline Insights engineering teammates. We currently have 1 open backend engineering position available if you are interested in applying. Almost 2/3s of the Pipeline Insights team has been focused on Verify Reliability initiatives related to Build Artifacts and availability. The Pipeline Insights team has not been able to execute on the planned direction as a result of this shift in global priorities. The Pipeline Insights group is responsible for 6 categories: Performance Testing, Usability Testing, Code Testing, Review Apps, Build Artifacts, and Accessibility Testing; of which we have chosen to not invest in some of those categories. We have moved Code Quality over to Secure to align with the other scanning capabilities.
In broad strokes, the Pipeline Insights group expected to move two categories in Maturity in FY22, but did not accomplish this target. One is Code Quality move to Viable, which recently moved to Secure, and the other was Performance Testing move to Viable. The Pipeline Insights Group is also expected to deliver key functionality related to Code Testing including the Project Quality UI to support the Paid GMAU PI for Pipeline Insights, and this has been delayed until at least FY23-Q4 as a result of the Verify Reliability efforts.
In FY23-Q1, the Pipeline Insights group will be prioritizing our Verify Reliability efforts. In FY23-Q2 and beyond, we are expecting to focus on Build Artifacts and Review Apps. We have since pivoted our PI focus from Paid GMAU to a GMAU with the inclusion of Build Artifacts in the Pipeline Insights charter and expect to enhance our free user base by improving the management and experience of artifacts in GitLab. With the Verify Stage persona focus on Sasha and Devon, the Pipeline Insights Team will need to explore developer-related JTBDs for testing capabilities in GitLab and prioritize those experiences to deliver on being the leader.
The Pipeline Insights direction has two opportunities for acceleration, first is to increase community contributions for categories like Review Apps which have a strong user following and the second is increasing headcount. There are two proposals for increased investment which include an ask for 3 engineering headcount to add a category for Shift-Left Testing, which would keep us competitive in the market from a testing capabilities POV as well as ask for 1 backend engineer to support S1/S2 burndown, especially related to Build Artifacts. Without the additional investment for Shift-Left Testing, we are unable to compete effectively in the production readiness realm for enterprises. Without the additional headcount for S1/S2 burndown, the Pipeline Insights Group will be unable to execute fully on the Code Testing direction and improvements to further enhance Accessibility Testing, Performance Testing, and Review Apps will be delayed until FY24-Q1.
We appreciate your continued support and engagement in the Verify, Pipeline Insights group!
Thanks for reading, Jackie