Code testing and coverage ensure that individual components built within a pipeline perform as expected, and are an important part of a Continuous Integration framework. Our vision for this category is to make the feedback loop for developers as short as possible, eventually enabling users to go from first commit to code in production in only an hour with confidence.
Interested in joining the conversation for this category? Please join us in the issues where we discuss this topic and can answer any questions you may have. Your contributions are more than welcome.
This page is maintained by the Product Manager for Testing, James Heimbuck (E-mail)
We know that displaying the current code coverage value for a project is nice and displaying the value in a pipeline is valuable. We also know that without historical context it can be hard to understand what that value means today vs. yesterday. To solve this problem for users we are next going to add a csv download and a graph of the code coverage value(s) found in jobs.
This category is currently at the "Minimal" maturity level, and our next maturity target is "Viable" (see our definitions of maturity levels). Key deliverables to achieve this are included in these epics:
We may find in research that only some of the issues in these epics are needed to move the vision for this category maturity forward.
Many other CI solutions can consume standard JUnit test output to display insights natively like CircleCI or through a plugin like Jenkins.
There are new entries in the space utilizing ML/AI tech to optimize testing like Launchable and write test cases like Diffblue.
In order to stay remain ahead of these competitors we will continue to push forward to make unit test data visible and actionable in the context of the Merge Request for developers with JUnit reports and historical insights to identify flaky tests with issues like gitlab#33932.
Sales has requested more mature coupling of test results as apart of the Release process. This is captured in gitlab#32773 from the Release Management group's Release Evidence category.
The most popular issue in the Code Testing and Coverage category today is a request to see the code coverage badge on any branch which would solve common problem for users of long lived branches who do not have a view of the test coverage of those branches today.
Another popular issue in the category is an interesting request to introduce a Checks API. This is a good first step towards cleaning up the appearance of the MR widget as more data is added there over time.
The QA department has opened an interesting issue gitlab#14954, aimed at solving a problem where they have limited visibility into long test run times that can impact efficiency.
The top vision item is gitlab#3673 which will start to address the problem of flaky test results which cause developers to not trust test runs or force unnecessary reruns of tests. Both of those outcomes are undesirable and counter to our goal of minimizing the lead time of changes.
We are looking to provide a one stop place for CI/CD leaders with gitlab#199739. Quality is an important driver for improving our users ability to confidently track deployments with GitLab and so we are working next on a view of code coverage data over time for individual projects and data across a group's projects.
