Product Vision - Verify

The Verify stage of the DevOps pipeline covers the CI lifecycle as well as testing (unit, integration, acceptance, performance, etc.). Our mission is to help developers feel confident in delivering their code to production. Please read through to understand where we're headed and feel free to jump right into the discussion by commenting on one of the epics or issues, or by reaching out directly to our product manager leading this area: Brendan O'Leary (E-Mail | Twitter)

North Stars

Gateway to Operations

CI was one of the first "additions" to GitLab where there was a hotly contested debate about placing it into a single application. As the first stage in the Ops Department, Verify has the unique ability to tie together what developers are doing (every commit) with what operators need (reliable, tested deployments). Therefore, Verify will continue to strive to be a "shining city on a hill" of the tangible and emergent benefits of a single application, avoiding the pitfalls of legacy systems that optimize only for one stage of the DevOps lifecycle.

Speed & Scalablity

At the core, GitLab CI/CD was built from the ground up with speed and scalability in mind. This is reflected both in the product design and product vision (automated testing, testing all the things) To enable our users to go from monthly shipping releases to truly enabling continuous delivery, everything that we do must be concerned with the scalability and speed of continuous integration and testing.

Lovable solutions to complex problems

Our users are solving some of the hardest problems on the cutting edge of software development. The promise of GitLab overall is to provide a simple, lovable application for the entire DevOps lifecycle.
Many parts of that cycle - including Verify - are traditionally complex to solve. Our goal is to provide a lovable solution to these complex problems so that users focus on their code - not building a DevOps environment.

Categories

There are a few categories that are critical for success in this stage; each one is intended to represent what you might find as an entire product out in the market. If you have thoughts or questions on any of these, feel free to jump into the conversation in the vision epic.

Continuous Integration (CI)

Gain the confidence to ship at blistering speed and immense scale with automated builds, testing, and out-of-the-box security to verify each commit moves you forward.
Learn moreDocumentationVision

Code Quality

Automatically analyze your source code to surface issues and see if quality is improving or getting worse with the latest commit.
DocumentationVision

Performance Testing

Be confident in the performance of your changes by ensuring that they are validated against real world load scenarios.
DocumentationVision

System Testing

Modern software is often delivered as a collection of (micro)services to multiple clouds, rather than a single monolith to your own data center. Validating complex interactions to ensure reliability of the system as a whole is more important than ever.
Vision

Usability Testing

Testing to ensure usability flows work and are actually understandable and valuable to your users.
Vision

Accessibility Testing

Beyond being a compliance requirement in many cases, accessibility testing is the right thing to do.
Vision

Compatibility Testing

Compatibility testing is a broad discipline and includes things such as hardware testing for software that runs on different devices, as well as multi-cloud compatibility testing which is becoming more and more important in a cloud-based world where you don't want all your eggs in one basket.
Vision

Our User's Journey to DevOps Maturity

We're really taking the idea of bringing GitLab users on the CI/CD journey seriously, and have used the great model here for our inspiration (though we have modified it slightly and will continue to do so over time.) Often, when businesses invest in continuous integration, they can plateau once they have have the code building and tests running. However, to deploy effectively, we need to embrace code review, dependency assurance, security scanning and performance metrics for every commit. GitLab's vision for Verify is to help users accelerate their journey to DevOps maturity.

Maturity Model

  Advanced Level Intermediate Level Baseline Level Beginner Level
Process & Organization Self organised and cross functional: The team can solve any task, releases continuously, and is doing continuous improvement. DevOps! Pull-based process: Measurement and reduction of cycle time. Continuous focus on process improvement. Always production ready code. Agile 101: Improved communication with business. Releases after each iteration. Developers have access to production logs. Silo organisation: People who in some way depend on each others work do not have effective ways of working together. Infrequent releases. Developers do not have access to production logs.
Technology Loosely coupled architecture: It is easy to replace technology for the benefit of something better (Branch by abstraction). Simple technology: In depth knowledge about each technology; why it is used and how it works. All of the technologies used are easy to configure and script against. Technology that makes it simple to roll back and forth between database versions. Best-of-breed: Chooses technology stack based on what is best for each purpose. Preference for Open Source. Avoids products that causes vendor lock-in. "Enterprise" infrastructure: Technology that can only be configured via a GUI. Large "enterprise" suites that do not work well in tandem and don't deliver what was promised.
Quality Assurance All testing automated: Almost all testing is automated, also for non-functional requirements. Testing of infrastructure code. Health monitoring for applications and environments and proactive handling of problems. Automated functional tests: Automated acceptance and system tests. Tests are written as part of requirements specification. All stakeholders specify tests. Automated technical tests: Automated unit and integration tests. Manual testing: Test department. Testing towards the end, not continuously. Developers do not test.
Deployment Routines One-click deploy: Everybody (including the customer) can deploy with one click. 0-downtime deploy. Feedback on database performance and deployment for each release. Automated deploy: Same process for deploy to all environments. Feature toggling to switch on/off functionality in production. Release and rollback is tested. Database migration and rollback is automated and tested for each deploy. Database performance is monitored and optimised. Repeatable deploy: Documented and partially automated deploy. Database changes are scripted and versioned. Manual deploy: Deployments require many manual steps. Manual and unversioned database migrations.
Configuration Management Infrastructure as code: Fully automated provisioning and validation of environments. Orchestration of environments. Application configuration control: All application configuration in version control. The application is configured in one place. Self service of development- and test environments. Dependency control: Dependencies and libraries are defined in version control. Manual configuration: Manual configuration in each environment and on each server.
Build & Continuous Integration Build/deploy pipeline: Same binary is deployed to all environments. Continuous improvement and automation of repeating tasks. Optimised for rapid feedback and visualisation of integration problems. Continuous integration: Continuous integration of source code to mainline. All changes (code, configuration, environments, etc.) triggers the feedback mechanisms. Artifact repository. Reuse of scripts and tools. Generation of reports for the build. Builds that fail are fixed immediately. CI-server: Automation of builds/tests on CI server. Can recreate builds from source code. Manual routines: Manual routines for builds. Lack of artifact repository. Lack of reports.

Upcoming Focus

To achieve our goals in the CI domain, we're looking at making big investments over the medium term in several key areas, including the following. We've aligned each quarter in 2019 to a step along the journey where we'll focus on improving that part of the path, though that doesn't mean we aren't looking at the big picture the whole year - these just represent periods where we're giving these areas special attention.

What we hope to achieve with this sequencing is taking our existing capabilites, which meet the need of a variety of users but tend to focus on baseline/intermediate capability, and pushing them forward with advanced, deep features to drive the entire product forward in the first quarter. Then, over the remainder of 2019, we'll peel the onion back ensuring there's a breadcrumb trail for all users to reach that level of maturity (not just in these features, but for all advanced features in GitLab CI/CD.)

We have a strong focus heading into each quarter:

After Q4 we've of course already begun thinking about where the product is headed. Watch this space soon for updates on where we see the product going in 2020 and beyond.

Finally, it's important to mention that this is our vision on the product at this time. All of this can change any moment and should not be taken as a hard commitment, though we do try to keep things generally stable and not change things all the time.

Prioritization Process

In general, we follow the same prioritization guidelines as the product team at large. Issues will tend to flow from having no milestone, to being added to the backlog, to being added to this page and/or a specific milestone for delivery.

You can see our entire public backlog for Verify at this link; filtering by labels or milestones will allow you to explore. If you find something you're interested in, you're encouraged to jump into the conversation and participate. At GitLab, everyone can contribute!

Issues with the "direction" label have been flagged as being particularly interesting, and are listed in the sections below.

Direction

11.7 (2019-01-22)

11.8 (2019-02-22)

11.9 (2019-03-22)

11.10 (2019-04-22)

11.11 (2019-05-22)

12.0 (2019-06-22)

2019 Q3

2019 Q4

Other Interesting Items

There are a number of other issues that we've identified as being interesting that we are potentially thinking about, but do not currently have planned by setting a milestone for delivery. Some are good ideas we want to do, but don't yet know when; some we may never get around to, some may be replaced by another idea, and some are just waiting for that right spark of inspiration to turn them into something special.

Remember that at GitLab, everyone can contribute! This is one of our fundamental values and something we truly believe in, so if you have feedback on any of these items you're more than welcome to jump into the discussion. Our vision and product are truly something we build together!