Code testing and coverage ensure that individual components built within a pipeline perform as expected, and are an important part of a Continuous Integration framework. Our vision for this category is to make the feedback loop for developers as short as possible, eventually enabling users to go from first commit to code in production in only an hour with confidence.
Interested in joining the conversation for this category? Please join us in the issues where we discuss this topic and can answer any questions you may have. Your contributions are more than welcome.
This page is maintained by the Product Manager for Testing, James Heimbuck (E-mail)
Users are excited to have data and a visual indicator of the direction code coverage is trending in a project. For users that have dozens to thousands of projects though having an even higher level view is often required, especially in the enterprise. Today those users are still manually gathering code coverage data to roll up into anther tool for data presentation. To solve this problem for users we are next going to build code coverage data the group level.
The first deliverable to customers in this epic will be a data download of all coverage data in a group followed by usability enhancements and a graph of the average coverage in a group.
This category is currently at the "Minimal" maturity level, and our next maturity target is "Viable" (see our definitions of maturity levels). Key deliverables to achieve this are included in these epics:
We may find in research that only some of the issues in these epics are needed to move the vision for this category maturity forward. The work to move the maturity is captured and being tracked in this epic.
Many other CI solutions can consume standard JUnit test output to display insights natively like CircleCI or through a plugin like Jenkins.
There are new entries in the space utilizing ML/AI tech to optimize testing like Launchable and write test cases like Diffblue.
In order to stay remain ahead of these competitors we will continue to push forward to make unit test data visible and actionable in the context of the Merge Request for developers with JUnit reports and historical insights to identify flaky tests with issues like gitlab#33932.
Sales has requested a higher level view of testing and coverage data for both projects and groups from the Testing Group. A first step towards this will be the display of coverage data for groups.
The most popular issue in the Code Testing and Coverage category today is a request to see the code coverage badge on any branch which would solve common problem for users of long lived branches who do not have a view of the test coverage of those branches today.
The GitLab Quality team has opened an interesting issue, Provide API to retrieve test case durations from a pipeline, that is aimed at solving a problem where they have limited visibility into long test run times that can impact efficiency.
The top vision item is Detect and report on flaky tests which will start to address the problem of flaky test results which cause developers to not trust test runs or force unnecessary reruns of tests. Both of those outcomes are undesirable and counter to our goal of minimizing the lead time of changes. The Testing team had a good discussion about this as part of a Think Big session in July, 2020. The very first step towards solving this problem is to make the duration data of tests available for anlaysis.
We are also looking to provide a one stop place for CI/CD leaders with Director-level CI/CD dashboards. Quality is an important driver for improving our users ability to confidently track deployments with GitLab and so we are working next on a view of code coverage data over time for across a group's projects.
