Reliability Engineering - How We Work: Issue Management and Prioritization Process

1. You are here:
2. Engineering
3. Infrastructure
4. Team
5. Reliability Engineering
6. Reliability Engineering - How We Work: Issue Management and Prioritization Process

Maintained by:

On this page

• Overview
• Issue Priority
  • Impact
  • Urgency
  • Priority
  • Service Level Agreements
• Classifying work
  • Corrective Actions and Security issues
  • Labels
    • Workflow Labels
    • Service Labels
      • Examples
    • Team Labels
    • T-shirt Labels
    • Other Labels
• Issue Review
• Issue Triage
  • 1. General Triage
  • 2. General Issue Prioritization
    • General Prioritization Guidelines
  • 3. Corrective Action Issue Review and Prioritization
    • Refinement of open Corrective Actions
    • Working on CAs
  • 4. External Issue Review
• Capacity Planning Warnings
• Epics

Overview

All requests for work to the Reliability Team come through the Reliability Issue Tracker. The management of this queue is an ongoing maintenance task for Reliability Engineers and Managers. This page contains an overview of the criteria used in determining how work is triaged and prioritized.

Issue Priority

Priority for incoming work is based on a matrix measuring the impact and urgency of an issue.

Impact

Impact is the measure of the effect of an incident, problem, or change on business processes as detailed in the issue.

The table below can be used as a general guide for determining impact:

Urgency

Urgency is the speed at which an issues should be resolved based on business need or expectation.

The table below can be used as a general guide for determining impact:

Priority

Once the impact and urgency of an issue has been determined, it is time to assign a priority.

The table below can be used as a general guide for assigning priority:

Note: Issue priority levels generally align with the severity levels defined within the Incident Management process.

Service Level Agreements

The Reliability team uses the following response time SLAs for all ~Reliability::P1 and ~Reliability::P2 issues that also have the ~unblocks others label. The ~unblocks others label indicates that the request originated from outside of the Infrastructure Department.

Note: Except for corrective actions and security issues, reliability does not use ~"priority" or ~"severity" labels, if these labels are added they will be removed.

Classifying work

Corrective Actions and Security issues

Corrective actions and security issues labeled as ~"corrective action" or ~"security" are labeled differently in the Reliability issue tracker. For these issues, severity::* labels are set to meet specific SLOs.

• For security issue see the Time to resolve table by severity
• Corrective Actions SLOs are currently based on definitions from Quality:

Note: For issues labeled with ~"security" and ~"corrective action" we do not use the ~"priority::*" labels, if added they will be removed. Please use the Reliability::P* labels instead.

Labels

Workflow Labels

Issues and epics should always use one of the following workflow labels:

Service Labels

service:: labels are used to route issues to the right team within Reliability. All issues should have a service:: label specified during the triage process. If multiple services are involved choose the service that best fits based on the details in the issue.

Examples

Team Labels

During the traige process, issues will be routed to a Reliability Team where appropriate. See the table below for the current list of active team Labels:

T-shirt Labels

T-shirt labels are used to estimate the size of issues. These are always a rough estimate and often need to be adjusted once the full scope of an issue is defined.

Other Labels

• unblocks others - Apply this label to all issues that originate outside of the Infrastructure Department

Issue Review

The issue board for Reliability is reviewed twice a week by Reliability Leadership Team. If you have an urgent issue that you believe should be prioritized ahead of other work, please reach out to any Engineering Manager on the Reliability Team to discuss.

Issue Triage

Issue Triage is performed twice weekly by members of the Reliability Leadership Team. The process consists of four parts.

1. General Triage

1. Locate issues that need to be triaged using this view which shows workflow-infra::Triage tickets lists most recently updated issues first.
2. Choose the first issue in the list and ask the following questions:
   1. For issues where the team ownership of the involved service is clear, choose a Reliability Team to assign the issue to and apply the appropriate team label.
   2. What is the priority of the issue? Read the issue and assign a priority as defined by our process. Once priority has been determined, add the appropriate priority label. NOTE: Prioritized issues are reviewed by Reliability Leadership multiple times a week, so it is OK if an issue is prioritized incorrectly.
3. Add the workflow-infra::Ready label. This indicates that the issue has been through the prioritization process and is ready to be looked at by an engineer.
4. Return the workflow-infra::Triage list view, select the next issue in the list and put it through the same process.
5. Add a service:: label.

2. General Issue Prioritization

The overall backlog of general issues is reviewed to assess and adjust priority. The following boards are used to help with this process:

1. Overall Priority of all triaged Reliability Issues.
2. All unassigned issues by priority
3. All issues in progress - review to ensure all issues with the ~workflow-infra::In Progress are truly active.
4. All assigned issues by priority

General Prioritization Guidelines

• Issues should be prioritized by their priority label first
• When faced with multiple issues of the same priority level:
  • prioritize small issues ahead of large ones.
  • prioritize older issues over new.

3. Corrective Action Issue Review and Prioritization

Corrective Actions for all of Infrastructure are reviewed, prioritized, and assigned to prevent or reduce the likelihood and/or impact of an incident recurrence. We use this board to track corrective actions work. Corrective Actions are also an important performance indicator for the Infrastructure Department.

1. All Open Infrastructure Corrective Actions by Severity
2. All Open Reliability Corrective Actions by Priority
   1. Observability
   2. Foundations
   3. Gitaly Stable Counterpart
   4. CI Runners Stable Counterpart
   5. Datastores::Postgres

Refinement of open Corrective Actions

The process is as follows:

1. Choose an open, unrefined "corrective action" issue (ie it does not have a "ca::refined label") in the GitLab Infrastructure Team project. Add the "workflow-infra::Under Review" label, and assign it to yourself. This ensures that other engineers don't pick up the same issue for refinement asynchronously. Issues with the label "ca::triage" or any of "severity::1", "severity::2", "priority::1", "priority::2" should be prioritized for refinement. This list can be a useful place to start.
2. Read the issue carefully, including any linked incidents, to get context.
3. If there is no linked incident, this issue may be mislabeled. Confirm with the issue creator and if there is no incident related to this issue, remove the "corrective action" label. In this case you are done refining this issue.
4. Ensure it has a severity label on it - this is the highest severity of linked incident(s).
5. Ensure it has a priority label on it, indicating the urgency of the issue. For guidance on priority level, see the engineering triage handbook page. When in doubt, feel free to ask the issue creator or other relevant engineers for input.
6. Based on the above, possible actions are:
   1. Cancel the issue by adding the label "workflow-infra::Cancelled" and then closing the issue
      1. Because it is no longer relevant due to other changes in the infrastructure, whether already executed or planned.
      2. You consider it an infrequent enough or low priority problem that you don't think it is currently worth prioritizing the work given the team's current resources.
      3. In either case, add a note to the issue explaining the reason for cancelling.
      4. Don't hesitate to cancel old, low priority CAs. An issue can always be reopened if it is deemed to be an ongoing problem, in which case we can increase the severity/priority and reassess.
   2. Indicate that this CA is ready for work by adding the label "workflow-infra::Ready"
      1. This indicates that the issue is well scoped (to be able to be completed with at most a few days' engineering time), and has the necessary information and context for an engineer to pick up for work.
   3. Promote to an Epic
      1. Sometimes CAs are high priority but very large in scope (more than a few days' worth of work). In this case the CA can be promoted to an Epic. Update the description of the new epic to show that it was created from a CA and make sure that the incident is referenced and linked in that paragraph.
      2. If it falls within the CA's scope, leave the Corrective Action label in place. Otherwise remove it with a note.
      3. New issues that are created related to an Epic with a "corrective action" label should not get that label, so as not to artificially inflate the numbers for tracking purposes.
   4. Move to another team/project
      1. Sometimes it becomes clear that a particular corrective action is not an Infrastructure issue. Add a note and move the issue to the appropriate project. If the issue is the responsibility of a stage group, consider adding the ~infradev label.
   5. Needs more information
      1. If context is missing or scope is unclear, ping the relevant engineers or managers in the issue to gain clarity so that a determination can be made for one of the above four cases. Sometimes we bring issues we need more input on up for discussion at one of our weekly Reliability Discussion meetings for this purpose.
   6. Mark the issue as "workflow-infra::Ready" and add the label "ca::refined" to indicate the Reliability General Team has looked at it. This indicates the issue is A. still relevant B. well scoped C. has enough context for the engineer picking up the issue and D. is ready for work.
7. When refinement is complete and the issue has one of above outcomes, label the issue with "ca::refined" to indicate refinement is completed and remove the "workflow-infra::Under Review" label. Unassign yourself from the issue unless you know you will be the one taking on the work.
8. Sometimes when refining a series of Corrective Action issues, a pattern may emerge that indicates that perhaps project level work is called for. Feel free to create an issue explaining your findings and bring this to the attention of the current DRI for corrective actions and/or an engineering manager.

Working on CAs

1. Choose a CA from the Ready column from the above linked board that has no engineer assigned to it. Start with the highest severity/priority level issue that you feel able to take on.
2. Assign yourself to the issue and move it to the "in progress" column or add the label "workflow-infra::In Progress".
3. Reach out to other engineers as needed for more context or pairing. CAs cover a very wide range of infrastructure areas, and you are not expected to know everything. Often it is much more efficient to pair with someone more knowledgable in that domain.
4. When the work is complete, remember to close the issue or move it to the "Done" column.

4. External Issue Review

1. All Active Incident Issues
2. Production Change Requests

Capacity Planning Warnings

Capacity planning warning issues are generated by the Scalability team and indicate concerns that we should address before they cause an alert (or an outage!). These issues are tracked outside of the Reliability project in the capacity planning project. The list of issues having the ~team::Reliability label is reviewed once per quarter and the work is then assigned based on impact and urgency.

• Current Reliability Capacity Planning Warning Issues

Epics

Note: Epics that require status tracking should be updated each Wednesday.

Example:

DRI: <!-- GitLab username for the DRI -->

## Status YYYY-MM-DD

<!-- Status summary, one or two sentences that is meant for management and team members outside of reliability.

     Example:

       ## Status 2022-02-02

       This week we rolled out the latest update to this amazing component, the team had a few setbacks that will be overcome in the next few weeks.
-->

### Additional status sections (optional)

<!-- Additional status updates underneath headings. For example, what was shipped, what is in progress, blockers, etc.
     This part won't be rolled up into the parent epic.

     Example:

       ### Details

        - Update `2.27.1` was rolled out to GPRD successfully.
        - Issue #44 was closed after the team agreed to cancel the work.
        - Issue #45 was completed successfully.
-->

### Overview

<!-- Overview of the project and its goals -->

### Reference (optional)

<!-- Links to boards, blueprints, readiness, or anything else relevant to the project -->