Guidelines are high-level directives on how we carry out operations and solve challenges in the Quality Engineering department.
Test Automation & Planning
Test plans as collaborative design document: Test Plans as documented in Test Engineering are design documents that aim to flush out optimal test coverage. It is expected that engineers in every cross-functional team take part in test plan discussions.
E2E test automation is a collective effort: Test Automation Engineers should not be the sole responsible party that automates the End-to-end tests. The goal is to have engineers contribute and own coverage for their teams.
We own test infrastructure: Test infrastructure is under our ownership, we develop and maintain it with an emphasis on ease of use, ease of debugging, and orchestration ability.
FutureDisable feature by default until E2E test merged: If a feature is to be merged without a QA test, it must be behind a feature flag (disabled by default) until a QA test is written and merged as well.
Flaky tests are quarantined until proven stable: A flaky test is as bad as no tests or in some cases worse due to the effort required to fix or even re-write the test. As soon as detected it is quarantined immediately to stabilize CI, and then fixed as soon as possible, and monitored until it is fixed.
Close issue when the test is moved out of quarantine: Quarantine issues should not be closed unless tests are moved out of quarantine.
Quarantine issues should be assigned and scheduled: To ensure that someone is owning the issue, it should be assigned with a milestone set.
Make relevant stage group aware: When a test fails no matter the reason, an issue should be created and made known to the relevant product stage group as soon as possible. In addition to notifying that a test in their domain fails, enlist help from the group as necessary.
Failure due to bug: If a test failure is a result of a bug, link the failure to the bug issue. It should be fixed as soon as possible.
Everyone can fix a test, the responsibility falls on the last who worked on it: Anyone can fix a failing/flaky test, but to ensure that a quarantined test isn't ignored, the last engineer who worked on the test is responsible for taking it out of quarantine.
~P1: Tests that are needed to verify fundamental GitLab functionality.
~P2: Tests that deal with external integrations which may take a longer time to debug and fix.
Triage packages to triage at scale: Due to the volume of issues, one team cannot handle the triage process. We have invented Triage Packages to scale the triage process within Engineering horizontally.
For test automation changes, it is crucial that every change is reviewed by at least one Senior Test Automation Engineer in the Quality team.
We are currently setting best practices and standards for Page Objects and REST API clients. Thus the first priority is to have test automation related changes reviewed and approved by the team. For test automation only changes, the quality team alone is adequate to review and merge the changes.
Building as part of GitLab
GitLab features first: Where possible we will implement the tools that we use as GitLab features.
Build vs buy: If there is a sense of urgency around an area we will consider buying/subscribing to a service to solve our Quality challenges in a timely manner. This is where building as part of GitLab is not immediately viable.