Guidelines are high-level directives on how we carry out operations and solve challenges in the Quality Engineering department.
Submitting and Reviewing code
For test automation changes, it is crucial that every change is reviewed by at least one Senior Test Automation Engineer in the Quality team.
We are currently setting best practices and standards for Page Objects and REST API clients. Thus the first priority is to have test automation related changes reviewed and approved by the team.
For test automation only changes, the quality team alone is adequate to review and merge the changes.
Test Automation & Planning
Test plans as collaborative design document: Test Plans as documented in Test Engineering are design documents that aim to flush out optimal test coverage.
It is expected that engineers in every cross-functional team take part in test plan discussions.
E2E test automation is a collective effort: Test Automation Engineers should not be the sole responsible party that automates the End-to-end tests.
The goal is to have engineers contribute and own coverage for their teams.
We own test infrastructure: Test infrastructure is under our ownership, we develop and maintain it with an emphasis on ease of use, ease of debugging, and orchestration ability.
FutureDisable feature by default until E2E test merged: If a feature is to be merged without a QA test,
it must be behind a feature flag (disabled by default) until a QA test is written and merged as well.
Flaky tests are quarantined until proven stable: A flaky test is as bad as no tests or in some cases worse due to the effort required to fix or even re-write the test.
As soon as detected it is quarantined immediately to stabilize CI, and then fixed as soon as possible, and monitored until it is fixed.
Close issue when the test is moved out of quarantine: Quarantine issues should not be closed unless tests are moved out of quarantine.
Quarantine issues should be assigned and scheduled: To ensure that someone is owning the issue, it should be assigned with a milestone set.
Make relevant stage group aware: When a test fails no matter the reason, an issue should be created and made known to the relevant product stage group as soon as possible.
In addition to notifying that a test in their domain fails, enlist help from the group as necessary.
Failure due to bug: If a test failure is a result of a bug, link the failure to the bug issue. It should be fixed as soon as possible.
Everyone can fix a test, the responsibility falls on the last who worked on it: Anyone can fix a failing/flaky test, but to ensure that a quarantined test isn't ignored,
the last engineer who worked on the test is responsible for taking it out of quarantine.
~P1: Tests that are needed to verify fundamental GitLab functionality.
~P2: Tests that deal with external integrations which may take a longer time to debug and fix.
Building as part of GitLab
GitLab features first: Where possible we will implement the tools that we use as GitLab features.
Build vs buy: If there is a sense of urgency around an area we may consider buying/subscribing to a service to solve our Quality challenges in a timely manner.
This is where building as part of GitLab is not immediately viable. An issue will be created to document the decision making process in our team task issue tracker.
This shall follow our dogfooding process.
Test Failure Management Rotation
This is a schedule to share the responsibility of debugging/analysing the failures in the daily runs in:
Please refer to the Quality team guidelines on debugging QA pipeline test failures for specific instructions on how to do an appropriate level of investigation and determine next steps for the failing test.
During the scheduled dates, the support tasks related to the test runs become the Directly Responsible Individual's (DRI's) highest priority.
Responsibilities of the DRI and Secondary for scheduled pipelines
The DRI does the triage and they let the counterpart TAE know of the failure.
The DRI makes the call whether to fix or quarantine the test.
The fix/quarantine MR should be reviewed by either the Secondary or the counterpart TAE (based on whoever is available). If both of them are not available immediately, then any other TAE can review the MR. In any case, both the Secondary and the counterpart TAE are always CC-ed in all communications.
The DRI should periodically take a look at the list of unassigned quarantined issues in staging and in nightly and work on them.
If the DRI is not available during their scheduled dates (for more than 2 days), they can swap their schedule with another team member. If the DRI's unavailability during schedule is less than 2 days, the Secondary can help with the tasks.
Responsibilities of the DRI and Secondary for deployment pipelines
The DRI helps the delivery team debug test failures that are affecting the release process.
The Secondary fills in when the DRI is not available.