Code testing and coverage ensure that individual components built within a pipeline perform as expected, and are an important part of a Continuous Integration framework. Our vision for this category is to make the feedback loop for developers as short as possible, eventually enabling users to go from first commit to code in production in only an hour with confidence.
Interested in joining the conversation for this category? Please join us in the issues where we discuss this topic and can answer any questions you may have. Your contributions are more than welcome.
This page is maintained by the Product Manager for Testing, James Heimbuck (E-mail)
Users are excited to have data and a visual indicator of the direction code coverage is trending in a project. For users that have dozens to thousands of projects though having an even higher level view of coverage is often required, especially in the enterprise. Providing a data download of all coverage data in a group to reduce manual work was a good first step.
The next steps are to display select data in GitLab for selected projects and a graph of the average coverage over all projects which will close out the work for Code Coverage for groups.
This category is currently at the "Minimal" maturity level, and our next maturity target is "Viable" (see our definitions of maturity levels). Key deliverables to achieve this are included in these epics:
We may find in research that only some of the issues in these epics are needed to move the vision for this category maturity forward. The work to move the maturity is captured and being tracked in this epic.
Many other CI solutions can also consume standard JUnit test output or other formats to display insights natively like CircleCI or through a plugin like Jenkins.
There are new entries in the code testing space utilizing ML/AI tech to optimize test execution like Launchable and even write test cases like Diffblue.
In order to stay remain ahead of these competitors we will continue to push forward to make unit test data visible and actionable in the context of the Merge Request for developers with unit test reports and historical insights to identify flaky tests with issues like gitlab#33932.
Sales has requested a higher level view of testing and coverage data for both projects and groups from the Testing Group. A first step towards this will be the display of coverage data for groups.
The most popular issue(s) in the Code Testing and Coverage category today are requests to be able to enforce a code coverage increase or hold or a coverage percentage before a merge.
Another popular issue is a request to see the code coverage badge on any branch which would solve common problem for users of long lived branches who do not have a view of the test coverage of those branches today.
The GitLab Quality team has opened an interesting issue, Provide API to retrieve test case durations from a pipeline, that is aimed at solving a problem where they have limited visibility into long test run times that can impact efficiency.
The top vision item is Detect and report on flaky tests which will start to address the problem of flaky test results which cause developers to not trust test runs or force unnecessary reruns of tests. Both of those outcomes are undesirable and counter to our goal of minimizing the lead time of changes. The Testing team had a good discussion about this as part of a Think Big session in July, 2020. The next step towards this are implementation of test history on the Test Summary Widget and Unit Test Report to learn if there is value in tracking and displaying test failures.
We are also looking to provide a one stop place for CI/CD leaders with Director-level CI/CD dashboards. Quality is an important driver for improving our users ability to confidently track deployments with GitLab and so we are working next on a view of code coverage data over time for across a group's projects.
We have started brainstorming some ideas for the vision and captured that as a rough design idea you can see below.
