Quality Engineering On-call Rotation

1. You are here:
2. Engineering
3. Quality Department
4. Quality Engineering Sub-Department
5. Quality Engineering On-call Rotation

On this page

• Quality Department pipeline triage on-call rotation
  • Responsibility
    • Triage, record, and unblock failures
    • Other recurring tasks
  • Schedule
  • Responsibilities of the DRI and Secondary for scheduled pipelines
  • Responsibilities of the DRI and Secondary for deployment pipelines
  • New hire on-call shadowing
• Quality Department incident management on-call rotation
  • Responsibility
  • Schedule

Quality Department pipeline triage on-call rotation

This is a schedule to share the responsibility of debugging/analysing the failures in the various scheduled pipelines that run on multiple environments. Quality department's on-call does not include work outside GitLab's normal business hours. Weekends and Family and Friends Days are excluded as well. In the current iteration, we have a timezone based rotation and triage activities happen during each team member's working hours.

Please refer to the Debugging Failing tests guidelines for an exhaustive list of scheduled pipelines and for specific instructions on how to do an appropriate level of investigation and determine next steps for the failing test.

Responsibility

Triage, record, and unblock failures

• The Saturday before the new rotation begins, the handover bot will create an issue in the pipeline-triage project.
• The handover bot also assigns the triage issue created to the Primary and Secondary DRIs for the upcoming week, according to the schedule below.
• During the scheduled dates, the support tasks related to the test runs become the Directly Responsible Individual's (DRI's) highest priority.
• Reporting and analyzing the End to End test failures in Staging, Canary and Production pipelines takes priority over GitLab master, GitLab FOSS master,and Nightly. Note this also takes priority over fixing tests.
• If there is a time constraint, the DRI should report and analyze the failures in Staging, Canary and Production pipelines just enough to determine if it is an application or an infrastructure problem, and escalate as appropriate. All the reported failures in those pipelines should be treated as ~priority::1/~severity::1 until it's determined that they're not. That means they should be investigated ASAP, ideally within 2 hours of the report. If the DRI will not be able to do so, they should delegate any investigation they're unable to complete to the Secondary. If the Secondary is not available or will also not be able to complete the investigations, solicit help in the #quality slack channel.
• Consider blocking the release by creating an incident if new ~severity::1 regressions are found in non-reliable/smoke specs.
• It is important that all other failure investigations are completed in a timely manner, ideally within 24 hours of the report. If the DRI is unable to investigate all the reported failures on all the pipelines on time, they should delegate the remaining work to the Secondary. If the Secondary is not available, solicit help in the #quality slack channel.
• Cross-cutting issues such as https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/530 are triaged by the on call DRI to determine the next action. Other team-members should notify the DRI if they come across such an issue. The DRI can inform the rest of the department via the #quality channel, if necessary.
• Everyone in the Quality department should support the on-call DRI and be available to jump on a zoom call or offer help if needed.
• Every day the APAC/EMEA/AMER Primary DRI summarizes the failures found for that day in the triage issue like this. This serves as an asynchronous handoff to the DRI in the other timezone.
• The DRI can make use of the dri gem to automate reporting handoff and surface useful triage information, such as newly created failures or feature flags enabled by environment.
• Be aware that failing reliable and smoke specs in staging/canary will block the deployer pipeline and may be treated as production incidents.

Other recurring tasks

• There is a weekly reminder in the #quality-weeklystandup channel to check if a new Chrome version was released and update pipelines if necessary. One of the current DRIs should take this on.
  • Merge request example: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/52008

Schedule

May 2022 - September 2022

Start Date	DRI	Secondary
2022-05-02	AMER: Erick Banks EMEA: Will Meek	APAC: John McDonnell EMEA: Andrejs Cunskis
2022-05-09	APAC: John McDonnell EMEA: Andrejs Cunskis	AMER: Sean Gregory APAC: Mark Lapierre
2022-05-16	AMER: Sean Gregory APAC: Mark Lapierre	AMER: Richard Chong APAC: Harsha Muralidhar
2022-05-23	AMER: Richard Chong APAC: Harsha Muralidhar	AMER: Désirée Chevalier APAC: Anastasia McDonald
2022-05-30	AMER: Désirée Chevalier APAC: Anastasia McDonald	AMER: Dan Davison EMEA: Sofia Vistas
2022-06-06	AMER: Dan Davison EMEA: Sofia Vistas	AMER: Valerie Burton EMEA: Nick Westbury
2022-06-13	AMER: Valerie Burton EMEA: Nick Westbury	AMER: Tiffany Rea EMEA: Andrejs Cunskis
2022-06-20	AMER: Tiffany Rea EMEA: Andrejs Cunskis	AMER: Zeff Morgan APAC: Sanad Liaquat
2022-06-27	AMER: Zeff Morgan APAC: Sanad Liaquat	AMER: Chloe Liu APAC: Careem Ahamed
2022-07-04	AMER: Chloe Liu APAC: Careem Ahamed	AMER: Erick Banks EMEA: Grant Young
2022-07-11	AMER: Erick Banks EMEA: Grant Young	APAC: John McDonnell EMEA: Edgars Brālītis
2022-07-18	APAC: John McDonnell EMEA: Edgars Brālītis	APAC: Mark Lapierre EMEA: Will Meek
2022-07-25	APAC: Mark Lapierre EMEA: Will Meek	AMER: Sean Gregory APAC: Harsha Muralidhar
2022-08-01	AMER: Sean Gregory APAC: Harsha Muralidhar	AMER: Richard Chong APAC: Carlo Catimbang
2022-08-08	AMER: Richard Chong APAC: Carlo Catimbang	AMER: Désirée Chevalier APAC: Anastasia McDonald
2022-08-15	AMER: Désirée Chevalier APAC: Anastasia McDonald	AMER: Dan Davison EMEA: Sofia Vistas
2022-08-22	AMER: Dan Davison EMEA: Sofia Vistas	AMER: Valerie Burton EMEA: Nick Westbury
2022-08-29	AMER: Valerie Burton EMEA: Nick Westbury	AMER: Zeff Morgan EMEA: Andrejs Cunskis
2022-09-05	AMER: Zeff Morgan EMEA: Andrejs Cunskis	AMER: Tiffany Rea APAC: Sanad Liaquat
2022-09-12	AMER: Tiffany Rea APAC: Sanad Liaquat	AMER: Chloe Liu APAC: Careem Ahamed
2022-09-19	AMER: Chloe Liu APAC: Careem Ahamed	AMER: Erick Banks EMEA: Grant Young
2022-09-26	AMER: Erick Banks EMEA: Grant Young	APAC: John McDonnell EMEA: Edgars Brālītis

Responsibilities of the DRI and Secondary for scheduled pipelines

• The DRI does the triage and they let the counterpart SET know of the failure.
• The DRI makes the call whether to fix or quarantine the test.
• The DRI reviews any automated quarantine MRs and automated dequarantine MRs and decides whether to merge or close.
• The fix/quarantine MR should be reviewed by either the Secondary or the counterpart SET (based on whoever is available). If both of them are not available immediately, then any other SET can review the MR. In any case, both the Secondary and the counterpart SET are always CC-ed in all communications.
• The DRI should periodically take a look at the list of unassigned quarantined issues and work on them.
• If the DRI is not available during their scheduled dates (for more than 2 days), they can swap their schedule with another team member. If the DRI's unavailability during schedule is less than 2 days, the Secondary can help with the tasks.

Responsibilities of the DRI and Secondary for deployment pipelines

• The DRI helps the delivery team debug test failures that are affecting the release process.
• The Secondary fills in when the DRI is not available.

New hire on-call shadowing

To help a new hire prepare for their first on-call rotation, they should spend a few days during their first rotation as Secondary DRI shadowing the Primary DRI. When the new hire is Primary DRI the following week, their Secondary DRI should pair with the new hire for a couple of days to answer questions and help triage/debug test failures.

Quality Department incident management on-call rotation

This is a schedule to share the responsibility of monitoring, responding to, and mitigating incidents. Quality department's on-call does not include work outside GitLab's normal business hours. Weekends and Family and Friends Days are excluded as well. In the current iteration, incident management activities happen during each team member's working hours.

Responsibility

• The Quality Engineering Manager should ensure they have joined the Slack channel #incident-management.
• During the scheduled week, the on-call DRI should monitor the incident management channel, tracking, directly helping, delegating, and raising awareness of incidents within the Quality Department or Quality Engineering Sub-Department as appropriate.
• Incidents should be passed along to the QEM of the stage impacted.
  • If the stage QEM is available, then the stage QEM becomes DRI for that specific incident.
  • If the stage QEM is unavailable, the on-call DRI maintains ownership of the specific incident until/unless the stage DRI becomes available.
• The current Quality DRI should be clearly noted on the incident issue.
• If a corrective action is needed, the DRI should create an issue and ensure it is labelled with ~'corrective action'.
• Everyone in the Quality Engineering Sub-Department should support the on-call DRI and be available to jump on a Zoom call or offer help if needed.

Schedule

April 2022 - July 2022

Start Date	DRI
2022-04-04	Ramya Authappan
2022-04-11	Vincy Wilson
2022-04-18	Joanna Shih
2022-04-25	Ramya Authappan
2022-05-02	Vincy Wilson
2022-05-09	Joanna Shih
2022-05-16	Ramya Authappan
2022-05-23	Vincy Wilson
2022-05-30	Joanna Shih
2022-06-06	Ramya Authappan
2022-06-13	Vincy Wilson
2022-06-20	Joanna Shih
2022-06-27	Ramya Authappan
2022-07-04	Vincy Wilson
2022-07-11	Joanna Shih
2022-07-18	Ramya Authappan
2022-07-25	Vincy Wilson

You can view the full Quality Department incident management on-call rotation on PagerDuty.