Gitlab hero border pattern left svg Gitlab hero border pattern right svg

Category Direction - System Testing

System Testing

Modern software is often delivered as a collection of (micro)services to multiple clouds, rather than a single monolith to your own data center. Validating complex interactions/compatibility in order to ensure the reliability of the system as a whole is more important than ever.

Interested in joining the conversation for this category? Please join us in our public epic where we discuss this topic and can answer any questions you may have. Your contributions are more than welcome.

This page is maintained by the Product Manager for Testing, James Heimbuck (E-mail)

What's Next & Why

Next up will be compatibility testing via gitlab#6061, which will introduce Selenium integration to capture multi-browser testing results. This solves a problem of testers today having to pull screenshots and logs after Selenium tests fail from artifact directory and directly displaying them in the job view.. This will start the ball rolling forward on how we can better support these kinds of cross browser testing needs.

We are also looking at a proof of concept of using Docker Compose to run integration tests for a project or group of projects that consists of interdependent microservices as part of gitlab#15992.

From there, we will work with our internal customer to understand what, if any problems that solves with our current use of gitlab-qa

Maturity Plan

This category is currently at the "Planned" maturity level, and our next maturity target is Minimal (see our definitions of maturity levels). Key deliverables to achieve this are:

It is possible we will choose one of the two above items to reach minimal level, depending on how user research plays out.

Competitive Landscape

No other CI platforms provide first-party compatibility or system testing, but all do provide different kinds of integrations. Selenium is a very popular one, and we plan to add a CI view for it via gitlab#6061. SauceLabs (gitlab#16034) is also a popular testing lab that provides devices for ensuring compatibility across a spectrum of browsers and/or devices.

API testing tools

API testing tools like SoupUI Pro, Apache JMeter and Hoverfly allow users to automate API and microservice testing across multiple services. There are integrations for CI/CD providers including GitLab for these tools but none offer native results within the pipeline today.

Top Customer Success/Sales Issue(s)

There are no top CS/Sales issues for this category.

Top Customer Issue(s)

The most popular item is (gitlab#16034), which integrates SauceLabs results from cross browser and platform tests.

Top Internal Customer Issue(s)

Apart from the CI view support for Selenium mentioned in the competitive landscape section, gitlab#16034 (integration support for SauceLabs) is also requested. team-tasks#45 tracks the progress of the internal quality team at GitLab rolling out compatibility testing.

Top Vision Item(s)

Adding CI view support for Selenium (gitlab#6061) is the most important vision item for the same reasons as in the competitive landscape above.

Another top vision issue for this category is group-level review apps.