Gitlab hero border pattern left svg Gitlab hero border pattern right svg

Triage Operations

On this page

Any GitLab team-member can triage issues. Keeping the number of un-triaged issues low is essential for maintainability, and is our collective responsibility.

We have implemented automation and tooling to handle this at scale and distribute the load to each team or group.

Video introduction to triage operations, triage report, priority and severity labels.

Accountability

The Quality Engineering Department ensures that every Product and Engineering group is held accountable to deliver on the SLA set forth.

Our defect SLA can be viewed at:

The Quality Engineering department employs a number of tools and automation in addition to manual intervention to help us achieve this goal. The work in this area can been seen in our department roadmap under Triage and Measure tracks of work.

Communicate early and broadly about expected automation impact

Be sure to give a heads-up in the relevant Slack channels (e.g. #development, #product) and the company call when an automation is expected to triage more than 100 issues or merge requests at once.

That is usually the case when migrating labels (e.g. migrating team labels to stage/group labels).

Auto-labelling of Issues

We currently auto label open issues based on the legacy team labels only. The mapping can be seen in Automation to ensure that issues and MRs with legacy team labels have a 1:1 mapping to their devops stage or group label.

Our triage bot will automatically infer stage and group labels based on the category/feature and team labels already set on an issue. This is available for open issues.

The most important rules are:

The following logic was initially implemented in this merge request:

graph TB; A{Stage label
is present?} -- Yes --> B; B{Group label
is present?} -- Yes --> X1[Nothing to do.]; B -- No --> E; E{Group is detected based on category labels
with a match rate > 50% among
all category labels?} -- Yes --> H; E -- No --> K; H{Does detected group label
matches stage label?} -- Yes --> X2[Set detected
group label.]; H -- No --> K; K{Several potential groups in
current stage detected
from category labels?} -- Yes --> X3[Manual triage
required.]; K -- No --> L; L{Does the stage has
a single group?} -- Yes --> X4[Set this
group label.]; L -- No --> X5[Manual triage
required.]; A -- No --> C; C{Group label
is present?} -- Yes --> X6[Set stage label based on
group label, we're done!]; C -- No --> G; G{Group is detected based on category labels
with a match rate > 50% among
all category labels?} -- Yes --> X7[Set group and
stage labels.]; G -- No --> X8[Manual triage
required.];

Check out the list of actual use-cases to better understand what this flow means in practice.

If your issue doesn't belong to a particular stage, you can remove the stage label and add the ~"automation:devops-mapping-disable" label to prevent this automation from happening in the future.

Auto-labelling of Merge Requests

We currently auto label open and merged merge requests based on the legacy team labels. The mapping can be seen in Automation to ensure that issues and MRs with legacy team labels have a 1:1 mapping to their devops stage or group label.

If your merge request doesn't belong to a particular stage, you can remove the stage label and add the ~"automation:devops-mapping-disable" label to prevent this automation from happening in the future.

Triage Packages

A triage package is an issue containing a checklist of issues requiring attention. Each task corresponds to an issue that needs labels, prioritization and/or scheduling.

Newly created unlabelled issues requiring first-triage

This package contains the 45 most recent unlabelled issues requiring initial triage. The goal is to ensure we achieve partial triage before the issue is picked up by a Product Manager and Engineering Manager in that area.

Group level bugs & features

This package contains the relevant bugs and feature requests that belong to a group in our DevOps stages. The goal is to achieve complete-triage by the Product Manager and Engineering Manager in that area.

The package itself is divided into 4 main parts.

The bug sections also contains a heatmap.

heatmap.png

An example: https://gitlab.com/gitlab-org/quality/triage-ops/issues/118

Video overview of the triage package.

There is also an optional stage policy for missing categories. If your team has enabled this, you will receive a list of up to 100 items that have the stage label but have zero appropriate category labels for that stage.

Feature proposals

This section contains issues with the ~"feature" label without a milestone. It is divided further into issues with and without ~"customer"

Frontend bugs

This section contains issues with the ~"bug" and ~"frontend" labels without priority and severity. It is divided further into issues with and without ~"customer"

Non-frontend bugs (likely backend)

This section contains issues with the ~"bug" label without priority and severity. It is divided further into issues with and without ~"customer"

P1 & P2 bugs past SLO

This section contains bugs which has past our targeted SLO based on the priority set. This is based on our missed SLO detection triage policy.

Community merge requests requiring attention

This package contains open merge requests which has been submitted by the wider community. These merge requests would have the ~"Community contribution" label.

The package itself is divided into 2 parts. The first part contains the 20 newest merge requests from the wider community. The second part contains 20 merge requests that weren't updated for 2 months or more.

Triage automation

General triage automation is run to label and update issues which help with reporting and milestone transition.

Milestone reschedule

Open issues and merge requests that have a closed milestone will be rescheduled to the next active milestone. This identifies pending work that was not completed within the planned milestone.

Missed deliverable

Identification of issues planned for a deliverable but missed will have a label applied. This enables reporting on issues which did not meet their release target.

Deliverable with no milestone

Issues which have a label of ~"Deliverable" without a milestone will have the milestone set to Backlog.

Missed SLO

Issues which have a priority and missed the SLO target will be labeled with ~"missed-SLO". The calculation for elapsed time starts from the date of the priority label being applied. This enables reporting on SLO target adherence.

Accepting merge requests

When milestone is present on an issue but there is not an assignee. The milestone being present indicates the product team has reviewed and scheduled the issue. This encourages open source contributions for planned features. Issues with the ~"workflow::blocked" label are excluded from this rule.

Master broken categorization

Issues or merge requests that have a label of ~"master:broken" will have labels of ~"P1" and ~"S1" applied. This ensures that requests which break master are sufficiently categorized for reporting.

Identify interesting feature proposals

This automation identifies potential and popular proposals using upvotes. This helps identify feature proposals that people have indicated they would like.

Community contributions

Merge requests which have an author that is not a member of gitlab-org will have a label of ~"Community contribution" applied. This informs the GitLab community team about new community contributions.

Resources