Gitlab hero border pattern left svg Gitlab hero border pattern right svg

Category Vision - System Testing

System Testing

Modern software is often delivered as a collection of (micro)services to multiple clouds, rather than a single monolith to your own data center. Validating complex interactions/compatibility in order to ensure the reliability of the system as a whole is more important than ever.

Interested in joining the conversation for this category? Please join us in our public epic where we discuss this topic and can answer any questions you may have. Your contributions are more than welcome.

What's Next & Why

Next up will be a proof of concept of using Docker Compose to run integration tests for a project or group of projects that consists of interdependent microservices as part of gitlab-ce#22559. From there, we will work with our internal customer to understand what, if any problems that solves with our current use of gitlab-qa

We're also looking at compatibility testing via gitlab-ee#6061, which will introduce Selenium integration to capture multi-browser testing results as well as screenshots and display them in a CI view. This will start the ball rolling forward on how we can better support these kinds of testing needs.

Maturity Plan

This category is currently at the "Planned" maturity level, and our next maturity target is Minimal (see our definitions of maturity levels). Key deliverables to achieve this are:

It is possible we will choose one of the two above items to reach minimal level, depending on how user research plays out.

Competitive Landscape

No other CI platforms provide first-party compatibility or system testing, but all do provide different kinds of integrations. Selenium is a very popular one, and we plan to add a CI view for it via gitlab-ee#6061. SauceLabs (gitlab-ce#22755) is also a popular testing lab that provides devices for ensuring compatibility across a spectrum of browsers and/or devices.

API testing tools

API testing tools like SoupUI Pro, Apache JMeter and Hoverfly allows users to automate API and microservices testing across multiple services.

Top Customer Success/Sales Issue(s)

There are no top CS/Sales issues for this category.

Top Customer Issue(s)

The most popular item is (gitlab-ce#22755), which integrates SauceLabs results from cross browser and platform tests.

Top Internal Customer Issue(s)

Apart from the CI view support for Selenium mentioned in the competitive landscape section, gitlab-ce#22755 (integration support for SauceLabs) is also requested. team-tasks#45 tracks the progress of the internal quality team at GitLab rolling out compatibility testing.

Top Vision Item(s)

The top vision issue for this category is group-level review apps.

Adding CI view support for Selenium (gitlab-ee#6061) is the most important vision item for the same reasons as in the competitive landscape above.