The following page may contain information related to upcoming products, features and functionality. It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes. Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features or functionality remain at the sole discretion of GitLab Inc.
Code testing and coverage ensure that individual components built within a pipeline perform as expected. This is a core piece of the Ops Section direction "Smart Feedback Loop" between developers and we are aiming to make that as reliably speedy as possible. Our long term vision is to enable users to go from first commit to code in production in only an hour with a high degree of confidence.
At GitLab we are building our features around JUnit.xml and Cobertura formatted reports primarily but have issues to expand beyond these in the future.
Check out our Ops Section Direction "Who's is it for?" for an in-depth look at our target personas across Ops. For Code Testing and Coverage, our "What's Next & Why" are targeting the following personas, as ranked by priority for support:
As part of a larger initiative to show a summary of the current state of a Project's Quality data the team will be building an endpoint to gather the latest test execution summary. We have found during solution interviews that engineering managers care the most about how many tests are passing and failing from the current data we can provide. This gives them an early indication if the project is ready to ship to customers in its current state.
Interested in joining the conversation for this category? Please join us in the issues where we discuss this topic and can answer any questions you may have. Your contributions are more than welcome.
This page is maintained by the Product Manager for Testing, James Heimbuck (E-mail)
This category is currently at the "Viable" maturity level, and our next maturity target is "Complete" (see our definitions of maturity levels). Key deliverables to achieve this are included in these epics:
We may find in research that only some of the issues in these epics are needed to move the vision for this category maturity forward. The work to move the maturity is captured and being tracked in this epic.
In the 2021 Continuous Software Delivery Forrester Tech Tide, Testing was cited as the number one key to unlock continuous delivery for organizations. Top areas for investment are a) API test automation, b) continuous functional test suites, c) shift-left performance testing. Industry leaders are seeking integrated suites over best in breed tools for testing and CD. Additionally, API testing is being marketed as a silver bullet that is cheaper, effective and efficient to modernize the toolchain for enterprises. Sample vendors include: API Fortress, Broadcom, Eggplant, and others. We are exploring how we expand our market share in this area via product#2516 and adding a new category in this merge request.
Many other CI solutions can also consume standard JUnit test output or other formats to display insights natively like CircleCI or through a plugin like Jenkins. Allure is a popular reporting tool for review of test executions and recently DataDog introduced CI Visibility as part of their SaaS offering including Flaky Test Management.
There are new entries in the code testing space utilizing ML/AI tech to optimize test execution like Launchable and even write test cases like Diffblue.
In order to stay remain ahead of these competitors we will continue to push forward to make unit test data visible and actionable in the context of the Merge Request for developers with unit test reports and historical insights to identify flaky tests with epics like gitlab&3129
Sales has requested a higher level view of testing and coverage data for both projects and groups from the Testing Group. We have started on solution validation with existing customers on designs in the Project Quality Summary epic.
Our two most popular customer requests are related to code coverage and code coverage reports.
Customers have quickly adopted and love the test coverage visualization feature feature but do not want to have to convert their coverage reports to the Cobertura. gitlab#227345 is a request to support JaCoCo coverage reports directly within the test coverage visualization feature. This better supports our vision of Speedy, Reliable Pipelines and our product principle of working by default.
For customers utilizing Git Flow getting test data like coverage and test summaries from features branches is hard to do and not easily surfaced at the project level. Customers have asked for a better way to display the test coverage of any branch in a badge.
The GitLab Quality team is interested in being able to detect and report on flaky tests. The MVC to count test failures was a good start and they are analyzing data from that feature to look at ways to improve.
The GitLab Quality team also opened an interesting issue, Provide API to retrieve test case durations from a pipeline, that is aimed at solving a problem where they have limited visibility into long test run times that can impact efficiency.
In 2020, Gartner has released the Artificial Intelligence Use Case Prism for Development and Testing on their research website. Directionally, several of the use cases are generation of unit tests from analyzing code patterns, using business logic to create API test scenarios, and using machine learning to fabricate test data as well as correlating testing results back to business metrics to convey meaningful connections like release success or quality.
To realize our long term vision we need to add more value not just for users uploading junit.xml and Cobertura reports but for any users with test and coverage reports. We believe that the best way to do this is to make it easy for users to contribute additional parsers so they can access the features the team is building that use the data. This will allow wider community contributions and is in alignment with GitLab's Dual Flywheel strategy. A first step towards this could be a GitLab-specific unit test report.
We are also looking to provide a one stop place for CI/CD leaders with Director-level CI/CD dashboards. Quality is an important driver for improving our users ability to confidently track deployments with GitLab and as noted above we are starting on that vision in the Project Quality Summary epic. We think this is a good first step towards the longer term vision shown in an exploratory design here.
