This page describes the process and technical documentation around reliable tests at GitLab, for both API and UI end-to-end tests located inside the
Reliable tests are executed as a blocking step in the release process. It is vital that these tests are optimized to run quickly and do not have transient failures. Transient failures of
reliable tests will lead to blocking the release team.
A reliable test is an end-to-end test that passes consistently in all pipelines, for at least 14 days. Such a test can be given the
These are the steps required to promote a new test to reliable
Note: the DRI for promoting new tests to reliable is the author of the MR that adds the new test(s). The author of the reliable test should consider adding documentation on how to run and debug the test prior to promoting it.
If an end-to-end test consistently passes for 14 consecutive days (as mentioned above), it could be considered a reliable test.
Note: the DRI for promoting existing tests to reliable is the author of the test. In case the author of the test isn't available, the counterpart SET of the test's DevOps stage should be the DRI.
Every week a reliable spec report is generated and report issue with the report summary is posted to slack channel
Test report contains all specs that have passed consecutively for 14 days on following pipelines:
Additionally, report contains information on
reliable specs that have failed in the past 14 days on any of the following pipelines:
Production Smoke and Reliable
Staging Smoke and Reliable
Staging Smoke and Reliable (No Admin)
A test is no longer considered reliable if it fails in any pipeline, including in merge requests, and the cause of the failure is found to be
In this case, the following process should be followed.
Note: A test is still reliable if it fails due to a bug in the application code, or due to issues with the application infrastructure that the test is not expected to handle.
Note 2: there's a detailed list of possible failures available in the debugging failing tests guideline, in the Classify and triage the test failure section
The following command is used to run the reliable tests:
bin/qa Test::Instance::All http://localhost:3000 -- --tag reliable
Note: in the above example,
http://localhost:3000 exemplifies how to run the reliable tests against a local GDK environment. This means that this argument can be changed to run the same tests against different environments.
Reliable tests will be run as part of the release process, during every deployment in staging, canary, and production environments.
This is in addition to the
smoke tests that is already run as part of the release process
If there are more suggestions/ open questions they can be added here too.