Quality Engineering On-call Rotation

1. You are here:
2. Engineering
3. Quality Department
4. Quality Engineering Sub-Department
5. Quality Engineering On-call Rotation

On this page

• Quality Department pipeline triage on-call rotation
  • Responsibility
    • Triage, record, and unblock failures
    • Other recurring tasks
  • Schedule
  • Responsibilities of the DRI for scheduled pipelines
  • Responsibilities of the DRI for deployment pipelines
  • New hire on-call shadowing
• Quality Department incident management on-call rotation
  • Responsibility
  • Schedule

Quality Department pipeline triage on-call rotation

This is a schedule to share the responsibility of debugging/analysing the failures in the various scheduled pipelines that run on multiple environments. Quality department's on-call does not include work outside GitLab's normal business hours. Weekends and Family and Friends Days are excluded as well. In the current iteration, we have a timezone based rotation and triage activities happen during each team member's working hours.

Please refer to the Debugging Failing tests guidelines for an exhaustive list of scheduled pipelines and for specific instructions on how to do an appropriate level of investigation and determine next steps for the failing test.

Responsibility

Triage, record, and unblock failures

• The Saturday before the new rotation begins, the handover bot will create an issue in the pipeline-triage project.
• The handover bot also assigns the triage issue created to the DRIs for the upcoming week, according to the schedule below.
• During the scheduled dates, the support tasks related to the test runs become the Directly Responsible Individual's (DRI's) highest priority.
• Reporting and analyzing the End to End test failures in Production, Canary, and Staging pipelines takes priority over GitLab master, GitLab FOSS master,and Nightly. Note this also takes priority over fixing tests.
• Preprod pipelines have equal priority with Production and Staging pipelines during release candidate testing from the 18th to the 22nd of each month.
• If there is a time constraint, the DRI should report and analyze the failures in Staging, Canary and Production pipelines just enough to determine if it is an application or an infrastructure problem, and escalate as appropriate. All the reported failures in those pipelines should be treated as ~priority::1/~severity::1 until it's determined that they're not. That means they should be investigated ASAP, ideally within 2 hours of the report. If the DRI will not be able to do so, they should delegate any investigation they're unable to complete to the DRI from the following week. If neither DRI is available or will be able to complete the investigations, solicit help in the #quality slack channel.
• Consider blocking the release by creating an incident if new ~severity::1 regressions are found in non-reliable/smoke specs.
• It is important that all other failure investigations are completed in a timely manner, ideally within 24 hours of the report. If the DRI is unable to investigate all the reported failures on all the pipelines on time, they should solicit help in the #quality slack channel.
• Cross-cutting issues such as https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/530 are triaged by the on call DRI to determine the next action. Other team-members should notify the DRI if they come across such an issue. The DRI can inform the rest of the department via the #quality channel, if necessary.
• Everyone in the Quality department should support the on-call DRI and be available to jump on a zoom call or offer help if needed.
• Every day the APAC/EMEA/AMER DRI summarizes the failures found for that day in the triage issue like this. This serves as an asynchronous handoff to the DRI in the other timezone.
• The DRI can make use of the dri gem to automate reporting handoff and surface useful triage information, such as newly created failures or feature flags enabled by environment.
• Be aware that failing reliable and smoke specs in staging/canary will block the deployer pipeline and may be treated as production incidents.

Other recurring tasks

• There is a weekly reminder in the #quality-weeklystandup channel to check if a new Chrome version was released and update pipelines if necessary. One of the current DRIs should take this on.
  • Merge request example: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/52008

Schedule

June 2022

Start Date	DRI	Secondary
2022-06-06	AMER: Dan Davison EMEA: Andrejs Cunskis	AMER: Valerie Burton EMEA: Nick Westbury
2022-06-13	AMER: Valerie Burton EMEA: Nick Westbury	AMER: Tiffany Rea EMEA: Andrejs Cunskis
2022-06-20	AMER: Tiffany Rea EMEA: Sofia Vistas	AMER: Zeff Morgan APAC: Sanad Liaquat
2022-06-27	AMER: Zeff Morgan APAC: Sanad Liaquat	AMER: Chloe Liu APAC: Careem Ahamed

July 2022 - October 2022

Start Date	APAC DRI	EMEA DRI	AMER DRI
2022-07-04	John McDonnell	Sofia Vistas	Richard Chong
2022-07-11	Mark Lapierre	Nick Westbury	Désirée Chevalier
2022-07-18	Harsha Muralidhar	Andrejs Cunskis	Dan Davison
2022-07-25	Anastasia McDonald	Grant Young	Valerie Burton
2022-08-01	Sanad Liaquat	Edgars Brālītis	Zeff Morgan
2022-08-08	Careem Ahamed	Will Meek	Tiffany Rea
2022-08-15	Carlo Catimbang	Alex Lyubenkov	Chloe Liu
2022-08-22	Erick Banks (AMER)	Nailia Iskhakova	Sean Gregory
2022-08-29	John McDonnell	Sofia Vistas	Richard Chong
2022-09-05	Mark Lapierre	Nick Westbury	Désirée Chevalier
2022-09-12	Harsha Muralidhar	Andrejs Cunskis	Dan Davison
2022-09-19	Anastasia McDonald	Grant Young	Valerie Burton
2022-09-26	Sanad Liaquat	Edgars Brālītis	Zeff Morgan
2022-10-10	Careem Ahamed	Will Meek	Tiffany Rea
2022-10-17	Carlo Catimbang	Alex Lyubenkov	Chloe Liu
2022-10-24	Erick Banks (AMER)	Nailia Iskhakova	Sean Gregory

Responsibilities of the DRI for scheduled pipelines

• The DRI does the triage and they let the counterpart SET know of the failure.
• The DRI makes the call whether to fix or quarantine the test.
• The DRI reviews any automated quarantine MRs and automated dequarantine MRs and decides whether to merge or close.
• The fix/quarantine MR should be reviewed by the counterpart SET. If the counterpart SET is not available immediately or if there is no counterpart SET, then any other SET can review the MR. In any case, the counterpart SET is always CC-ed in all communications.
• The DRI should periodically take a look at the list of unassigned quarantined issues and work on them.
• If the DRI is not available during their scheduled dates (for more than 2 days), they can swap their schedule with another team member. If the DRI's unavailability during schedule is less than 2 days, the DRI in the same timezone the following week can cover.

Responsibilities of the DRI for deployment pipelines

• The DRI helps the delivery team debug test failures that are affecting the release process.
• The DRI in the same timezone for the following week fills in when the DRI is not available.

New hire on-call shadowing

To help a new hire prepare for their first on-call rotation, they should spend a few days before their first rotation shadowing the DRI. When the new hire is DRI, the DRI from the following week should pair with the new hire for a couple of days to answer questions and help triage/debug test failures.

Quality Department incident management on-call rotation

This is a schedule to share the responsibility of monitoring, responding to, and mitigating incidents. Quality department's on-call does not include work outside GitLab's normal business hours. Weekends and Family and Friends Days are excluded as well. In the current iteration, incident management activities happen during each team member's working hours.

Responsibility

• The Quality Engineering Manager should ensure they have joined the Slack channel #incident-management.
• During the scheduled week, the on-call DRI should monitor the incident management channel, tracking, directly helping, delegating, and raising awareness of incidents within the Quality Department or Quality Engineering Sub-Department as appropriate.
• Incidents should be passed along to the QEM of the stage impacted.
  • If the stage QEM is available, then the stage QEM becomes DRI for that specific incident.
  • If the stage QEM is unavailable, the on-call DRI maintains ownership of the specific incident until/unless the stage DRI becomes available.
• The current Quality DRI should be clearly noted on the incident issue.
• If a corrective action is needed, the DRI should create an issue and ensure it is labelled with ~'corrective action'.
• Everyone in the Quality Engineering Sub-Department should support the on-call DRI and be available to jump on a Zoom call or offer help if needed.

Schedule

April 2022 - July 2022

Start Date	DRI
2022-04-04	Ramya Authappan
2022-04-11	Vincy Wilson
2022-04-18	Joanna Shih
2022-04-25	Ramya Authappan
2022-05-02	Vincy Wilson
2022-05-09	Joanna Shih
2022-05-16	Ramya Authappan
2022-05-23	Vincy Wilson
2022-05-30	Joanna Shih
2022-06-06	Ramya Authappan
2022-06-13	Vincy Wilson
2022-06-20	Joanna Shih
2022-06-27	Ramya Authappan
2022-07-04	Vincy Wilson
2022-07-11	Joanna Shih
2022-07-18	Ramya Authappan
2022-07-25	Vincy Wilson

You can view the full Quality Department incident management on-call rotation on PagerDuty.