The following page may contain information related to upcoming products, features and functionality. It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes. Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features or functionality remain at the sole discretion of GitLab Inc.
Code testing and coverage ensure that individual components built within a pipeline perform as expected. Another angle of code testing is accessibility. This is a core piece of the Ops Section direction "Smart Feedback Loop" between developers and we are aiming to make that as reliably speedy as possible. Our long term vision is to enable users to go from first commit to code in production in only an hour with a high degree of confidence.
At GitLab we are building our features around JUnit.xml and Cobertura formatted reports primarily but have issues to expand beyond these in the future.
Check out our Ops Section Direction "Who's is it for?" for an in-depth look at our target personas across Ops. For Code Testing and Coverage, our "What's Next & Why" are targeting the following personas, as ranked by priority for support:
There are no planned features in Code Testing and Coverage until 2023, though Pipeline Insights will support high priority bug fixes for this category in 2022.
We have completed the problem validation process on the opportunity "Detect and report flaky tests" &9183. In 15.8 we have started the solution validation process and anticipate completing our design in FY24-Q1. Due to the big impact for customers and high popularity in this area, this is our highest priority for Code Testing and Coverage.
Interested in joining the conversation for this category? Please join us in the issues where we discuss this topic and can answer any questions you may have. Your contributions are more than welcome.
This page is maintained by the Product Manager for Pipeline Insights, Jocelyn Eillis (E-mail).
This category is currently at the "Viable" maturity level, and our next maturity target is "Complete" (see our definitions of maturity levels). Key deliverables to achieve this are included in these epics:
In the 2021 Continuous Software Delivery Forrester Tech Tide, Testing was cited as the number one key to unlock continuous delivery for organizations. Top areas for investment are a) API test automation, b) continuous functional test suites, c) shift-left performance testing. Industry leaders are seeking integrated suites over best in breed tools for testing and CD. Additionally, API testing is being marketed as a silver bullet that is cheaper, effective and efficient to modernize the toolchain for enterprises. Sample vendors include: API Fortress, Broadcom, Eggplant, and others. We are exploring how we expand our market share in this area via product#2516 and adding a new category in this merge request.
Many other CI solutions can also consume standard JUnit test output or other formats to display insights natively like CircleCI or through a plugin like Jenkins. Allure is a popular reporting tool for review of test executions and recently DataDog introduced CI Visibility as part of their SaaS offering including Flaky Test Management.
There are new entries in the code testing space utilizing ML/AI tech to optimize test execution like Launchable and even write test cases like Diffblue.
In order to stay remain ahead of these competitors we will continue to push forward to make unit test data visible and actionable in the context of the Merge Request for developers with unit test reports and historical insights to identify flaky tests with epics like gitlab&3129
Sales has requested a higher level view of testing and coverage data for both projects and groups from the Pipeline Insights Group. We have started on solution validation with existing customers on designs in the Project Quality Summary epic.
Our two most popular customer requests are related to code coverage and code coverage reports.
Customers have quickly adopted and love the test coverage visualization feature feature but do not want to have to convert their coverage reports to the Cobertura. gitlab#227345 is a request to support JaCoCo coverage reports directly within the test coverage visualization feature. This better supports our vision of Speedy, Reliable Pipelines and our product principle of working by default.
For customers utilizing Git Flow getting test data like coverage and test summaries from features branches is hard to do and not easily surfaced at the project level. Customers have asked for a better way to display the test coverage of any branch in a badge.
The GitLab Quality team is interested in being able to detect and report on flaky tests. The MVC to count test failures was a good start and they are analyzing data from that feature to look at ways to improve.
The GitLab Quality team also opened an interesting issue, Provide API to retrieve test case durations from a pipeline, that is aimed at solving a problem where they have limited visibility into long test run times that can impact efficiency.
The Field and Quality Teams have expressed interest in Show code coverage diff as soon as coverage report is available, which is also a popular customer requested issue at 39 upvotes.
Additionally, the Design team has an open accessibility epic that contains items about making GitLab itself more accessible: gitlab-org&567. The long term vision for this category is for that team to be able to use GitLab to detect all of those issues in an automated way and see that they are addressed when fixed.
In 2020, Gartner has released the Artificial Intelligence Use Case Prism for Development and Testing on their research website. Directionally, several of the use cases are generation of unit tests from analyzing code patterns, using business logic to create API test scenarios, and using machine learning to fabricate test data as well as correlating testing results back to business metrics to convey meaningful connections like release success or quality.
To realize our long term vision we need to add more value not just for users uploading junit.xml and Cobertura reports but for any users with test and coverage reports. We believe that the best way to do this is to make it easy for users to contribute additional parsers so they can access the features the team is building that use the data. This will allow wider community contributions and is in alignment with GitLab's Dual Flywheel strategy. A first step towards this could be a GitLab-specific unit test report.
We also know that users want more insights from their CI/CD pipelines and especially from their tests. We are evaluating gitlab#210250 as a way to provide those insights and further encourage users to upload test report artifacts within their CI/CD pipelines.
We are also looking to provide a one stop place for CI/CD leaders with Director-level CI/CD dashboards. Quality is an important driver for improving our users ability to confidently track deployments with GitLab and as noted above we are starting on that vision in the Project Quality Summary epic. We think this is a good first step towards the longer term vision shown in an exploratory design here. As a end to end testing platform, enabling organizations to track how accessibility is changing over time by providing additional scanners will empower better decisions around code quality and accessibility.
