We use Fibonacci Series for weights and limit the highest number to 8. The definitions are as below:
1 - Trivial
Simple and quick changes (e.g. typo fix, test tag update, trivial documentation additions)
2 - Small
Straight forward changes, no underlying dependencies needed. (e.g. new test that has existing factories or page objects)
3 - Medium
Well understood changes with a few dependencies. Few surprises can be expected. (e.g. new test that needs to have new factories or page object / page components)
5 - Large
A task that will require some investigation and research, in addition to the above weights (e.g. Tests that need framework level changes which can impact other parts of the test suite)
8 - X-large
A very large task that will require much investigation and research. Pushing initiative level
13 or more
Please break the work down further, we do not use weights higher than 8.
Submitting and Reviewing code
For test automation changes, it is crucial that every change is reviewed by at least one Senior Software Engineer in Test (SET) in the Quality team.
We are currently setting best practices and standards for Page Objects and REST API clients. Thus the first priority is to have test automation related changes reviewed and approved by the team.
For test automation only changes, the Quality Department alone is adequate to review and merge the changes.
Test Automation & Planning
Test plans as collaborative design document: Test Plans as documented in Test Engineering are design documents that aim to flush out optimal test coverage.
It is expected that engineers in every cross-functional team take part in test plan discussions.
E2E test automation is a collective effort: SETs should not be the sole responsible party that automates the End-to-end tests.
The goal is to have engineers contribute and own coverage for their teams.
We own test infrastructure: Test infrastructure is under our ownership, we develop and maintain it with an emphasis on ease of use, ease of debugging, and orchestration ability.
FutureDisable feature by default until E2E test merged: If a feature is to be merged without a QA test,
it must be behind a feature flag (disabled by default) until a QA test is written and merged as well.
Flaky tests are quarantined until proven stable: A flaky test is as bad as no tests or in some cases worse due to the effort required to fix or even re-write the test.
As soon as detected it is quarantined immediately to stabilize CI, and then fixed as soon as possible, and monitored until it is fixed.
Close issue when the test is moved out of quarantine: Quarantine issues should not be closed unless tests are moved out of quarantine.
Quarantine issues should be assigned and scheduled: To ensure that someone is owning the issue, it should be assigned with a milestone set.
Make relevant stage group aware: When a test fails no matter the reason, an issue should be created and made known to the relevant product stage group as soon as possible.
In addition to notifying that a test in their domain fails, enlist help from the group as necessary.
Failure due to bug: If a test failure is a result of a bug, link the failure to the bug issue. It should be fixed as soon as possible.
Everyone can fix a test, the responsibility falls on the last who worked on it: Anyone can fix a failing/flaky test, but to ensure that a quarantined test isn't ignored,
the last engineer who worked on the test is responsible for taking it out of quarantine.
~P1: Tests that are needed to verify fundamental GitLab functionality.
~P2: Tests that deal with external integrations which may take a longer time to debug and fix.
Building as part of GitLab
GitLab features first: Where possible we will implement the tools that we use as GitLab features.
Build vs buy: If there is a sense of urgency around an area we may consider buying/subscribing to a service to solve our Quality challenges in a timely manner.
This is where building as part of GitLab is not immediately viable. An issue will be created to document the decision making process in our team task issue tracker.
This shall follow our dogfooding process.
Quality Department pipeline triage rotation
This is a schedule to share the responsibility of debugging/analysing the failures
in the various scheduled pipelines that run on multiple environments.
If there is a time constraint, the DRI should report and analyze the failures in Staging, Canary and Production pipelines just enough to determine if it is an application or an infrastructure problem, and escalate as appropriate. All the reported failures in those pipelines should be treated as ~P1/~S1 until it's determined that they're not. That means they should be investigated ASAP, ideally within 2 hours of the report. If the DRI will not be able to do so, they should delegate any investigation they're unable to complete to the Secondary. If the Secondary is not available or will also not be able to complete the investigations, solicit help in the #quality slack channel.
It is important that all other failure investigations are completed in a timely manner, ideally within 24 hours of the report. If the DRI is unable to investigate all the reported failures on all the pipelines on time, they should delegate the remaining work to the Secondary. If the Secondary is not available, solicit help in the #quality slack channel.
Everyday the APAC/EMEA Primary DRI summarizes the failures found for that day in the triage issue and this serves as an asynchronous handoff to the AMER DRI.
On Tuesdays and Thursdays at 8:30AM PST, there is a "Pipeline on-call rotation standup/hand-off" meeting which help in more collaboration.
Responsibilities of the DRI and Secondary for scheduled pipelines
The DRI does the triage and they let the counterpart SET know of the failure.
The DRI makes the call whether to fix or quarantine the test.
The fix/quarantine MR should be reviewed by either the Secondary or the counterpart SET (based on whoever is available). If both of them are not available immediately, then any other SET can review the MR. In any case, both the Secondary and the counterpart SET are always CC-ed in all communications.
If the DRI is not available during their scheduled dates (for more than 2 days), they can swap their schedule with another team member. If the DRI's unavailability during schedule is less than 2 days, the Secondary can help with the tasks.
Responsibilities of the DRI and Secondary for deployment pipelines
The DRI helps the delivery team debug test failures that are affecting the release process.
The Secondary fills in when the DRI is not available.
Engineering Productivity team rotation and escalation
In order to provide better input during the planning phase of a release, the schedule will include a member of the Engineering Productivity team during the second week of the month.
The Engineering Productivity team will use #quality to signal assistance in triaging a failure.
The Engineering Productivity team will cover the backup responsibility of the Secondary if the assigned Engineering Productivity Engineer (EPE) is not able to fulfill the responsibilities.