Any GitLab team-member can triage issues. Keeping the number of un-triaged issues low is essential for maintainability, and is our collective responsibility.
We have implemented automation and tooling to handle this at scale and distribute the load to each team or group.
Video introduction to triage operations, triage report, priority and severity labels.
The Quality Engineering Department ensures that every Product and Engineering group is held accountable to deliver on the SLA set forth.
Our defect SLA can be viewed at:
The Quality Engineering department employs a number of tools and automation in addition to manual intervention to help us achieve this goal. The work in this area can been seen in our department roadmap under Triage and Measure tracks of work.
There is a large amount of automation that uses stage, group, and category labels. We ask that Product Managers create an issue in triage-ops when any of the following changes occur. This issue helps ensure limited to no impact to automation and reports.
Our triage bot will automatically infer section, stage, and group labels based
on the category/feature already set on an issue or MR. This is available for open issues/MRs within the gitlab-org group.
The most important rules are:
stages.yml and the label is already set.The following logic was initially implemented in this merge request:
After the above inference is done, a section label will be added based on the stage or group label. An explanation will not be added in this step if the inferred labels contain only a section label.
Check out the list of actual use-cases to better understand what this flow means in practice.
If your issue/MR doesn't belong to a particular stage, you can remove the stage
label and add the ~"automation:devops-mapping-disable" label to prevent this
automation from happening in the future.
A triage report is an issue containing a checklist of issues or merge requests requiring attention. Usually, each task corresponds to an issue or a merge request that needs labels, prioritization, scheduling, attention etc. Some reports also include heatmaps or other various information.
Triage report are automatically assigned to specific team members, listed in the group definition file, or directly in the triage report policy files policy YAML files.
To change who an issue gets assigned to, open a merge request for the above files. If the group definition file is changed, we'll need to run some scripts to update the generated files as well.
These reports are owned by the Contributor Success team.
This report contains community merge requests requiring partial triage. The goal is for coaches to add type, stage, and group labels, so that the relevant people can be pinged later on based on these labels.
Community contribution which are not partially triaged.This report contains community merge requests that may require some attention from GitLab team members.
This is a daily generated report that contains the most recent unlabelled issues requiring initial triage. Currently, the number of issues indexed is based on the number of active Quality Department Engineering Managers for that day. This is 4 items per active Manager, so at most 16 items per day. The goal is to ensure we achieve partial triage before the issue is picked up by a Product Manager and Engineering Manager in that area.
Sometimes the unlabelled triage report creation fails due to spam detection. When the issue is not created, a notification for failure of the verify-unlabelled job will be raised to the #triage-automations channel.
The issue can be created manually using the following command:
bundle exec gitlab-triage --debug --r ./plugins/all.rb --token PERSONAL_API_TOKEN --source projects --source-id gitlab-org/gitlab -f ./policies/stages/report/unlabelled-issues.yml
This report contains the relevant bugs, feature requests, and UX debt issues that belong to a group in our DevOps stages. The goal is to achieve complete-triage by the Product Manager, Engineering Manager, UX team member in that area.
The report itself is divided into 4 main parts.
~priority::1 and ~priority::2 bugs past the target SLO.The bug sections also contains a heatmap.
An example: https://gitlab.com/gitlab-org/quality/triage-ops/issues/118
Video overview of the triage report.
There is also an optional stage policy for missing categories. If your team has enabled this, you will receive a list of up to 100 items that have the stage label but have zero appropriate category labels for that stage.
This section contains issues with the ~"type::feature" label without a milestone. It is divided further into issues with and without ~"customer"
Backlog or Awaiting further demand milestone.This section contains issues with the ~"type::bug" and ~"frontend" labels without priority and severity. It is divided further into issues with and without ~"customer"
Backlog.This section contains issues with the ~"type::bug" label without priority and severity. It is divided further into issues with and without ~"customer"
Backlog.This section contains bugs which has past our targeted SLO based on the severity label set. This is based on our missed SLO detection triage policy.
This section contains a table displaying the open issues for a group labelled with ~"customer" and ~"bug". There is a breakdown by the assigned severity and priority labels
This report contains idle group merge requests authored by GitLab team members.
Merge requests are considered idle when they have no human activity for 28 days. This report collects them for prompting of any actions to move the MR forward, such as nudging the author, reviewer, or maintainer.
An example report: Merge requests requiring attention for group::access - 2020-11-08. Current reports can be found in the triage-reports project
This report contains feature flags that have enabled in the codebase for 2 or more releases for groups within our DevOps stages.
The DRI is responsible for reviewing these feature flags to determine if they are able to be removed entirely, or create separate issues to ensure the overdue feature flags are removed accordingly.
An example report: Feature Flags requiring attention for group::continuous integration - 2021-03-01. Current reports can be found in the triage-reports project
The feature flag triage reports are generated in a quality toolbox scheduled pipeline with the gitlab-feature-flag-alert project.
Reports open for more than 2 weeks with the ~"triage report" label will be closed automatically with the close old triage reports automation.
Reactive triage automation is complementary to scheduled triage automation where realtime feedback provides an improved developer experience. This is handled by triage-ops.
gitlab-org group and its author is not a member of the gitlab-org group, or the MR is opened for the gitlab-com/www-gitlab-com project and its author is not a member of the gitlab-com group~"Community contribution" and ~"workflow::in dev"~"workflow::ready for review" was addedgitlab-org group and its author is not a member of the gitlab-org group~"workflow::ready for review" was addedgitlab-org group and its author is not a member of the gitlab-org groupCODEOWNERS file) to review~"workflow::ready for review" was addedgitlab-org group and its author is not a member of the gitlab-org group~UX is set#ux-community-contributions Slack channel (internal) to ask a UX reviewer to review non-draft MRshelp command@gitlab-bot help is posted on a merge requestgitlab-org group and its author is not a member of the gitlab-org groupready command@gitlab-bot ready, @gitlab-bot review, or @gitlab-bot request_reviewgitlab-org group and its author is not a member of the gitlab-org group~"workflow::ready for review" to the MRlabel command@gitlab-bot label ~"label-name" where label-name matches group::*, type::* or is ~"workflow::in dev" or ~"workflow::ready for review"gitlab-org group and its author is not a member of the gitlab-org groupgitlab-org group and its author is not a member of the gitlab-org group~idle or ~stale is set on the merge request~idle and ~stalegitlab-org group and its author is not a member of the gitlab-org group, or the MR is opened in a project under the gitlab-com group and its author is not a member of the gitlab-com groupfeedback command@gitlab-feedbackgitlab-org group and its author is not a member of the gitlab-org group#mr-feedback Slack channel (internal) pointing to the contributor feedback notegitlab-org group and its author is not a member of the gitlab-org group, or the MR is opened in a project under the gitlab-com group and its author is not a member of the gitlab-com group~Hackathon labelFor issues labelled ~"availability", the minimal are enforced with the
guidelines at
https://about.gitlab.com/handbook/engineering/quality/issue-triage/#availability-prioritization
Whenever ~"backstage [DEPRECATED]" is added, it'll remove it and hint
about why it should not be added, and alternatives will be provided.
The ~"customer" label is applied when a customer associated link is applied.
The following URLs are considered customer associated links:
gitlab.zendesk.comgitlab.my.salesforce.comWhenever a subtype label is added, the corresponding type label is added. Current type labels with subtype labels are:
~"type::feature"~"type::tooling"Scheduled triage automation is run to label and update issues which help with reporting and milestone transition. This is handled by triage-ops.
Accepting merge requests label to unassigned issues that have a milestoneWhen milestone is present on an issue but there is not an assignee. The milestone being present indicates the product team has reviewed and scheduled the issue. This encourages open source contributions for planned features.
Issues with the ~workflow::blocked, ~workflow::refinement, ~workflow::validation backlog, ~workflow::problem validation, ~workflow::solution validation, ~workflow::planning breakdown, ~workflow::design, ~workflow::verification, ~fuzzing::api, ~fuzzing::coverage labels are excluded from this rule.
~"Accepting merge request" label is applied.Seeking community contributions from issues with an assigneeWhen an issue is assigned, it shouldn't accept any new contribution to prevent duplicated work.
Seeking community contributions or Accepting merge requests label.Seeking community contributions and Accepting merge request labels are removed.Seeking community contributions from issues with an invalid workflow labelWhen an issue has the Seeking community contributions label set, but also an incompatible workflow label, the issue isn't actually ready to accept a contribution.
Seeking community contributions and one of the workflow::blocked, workflow::design, workflow::planning breakdown, workflow::refinement, workflow::verification labels.Seeking community contributions and Accepting merge request labels are removed.Seeking community contributions from all merge requestsIt doesn't make sense to have Seeking community contributions or Accepting merge request set on merge requests.
Seeking community contributions or Accepting merge requests label.Seeking community contributions and Accepting merge request labels are removed.Merge requests which have an author that is not a member of gitlab-org will have the Community contribution label applied. This scheduled automation is a backup for the reactive automation that applies Community contribution in the welcome message.
gitlab-org with author that is not in the gitlab-org group.Community contribution is applied, and optionally the 1st contribution label if it's the first contribution from the author in this project.Merged merge requests with the Community contribution label and no milestone will automatically get the relevant milestone set. This helps keep the community contributions numbers accurate.
Community contribution label, and no milestone.merged_at of the merge request and the start_date and due_date of the milestone.Community contribution label, and no human interaction for more than 28 days.idle label is applied.Community contribution label, and no human interaction for more than 120 days.stale label is applied.staleCommunity contribution and stale labels.Open issues and merge requests that have missed the current release will be rescheduled to the next active milestone. This identifies pending work that was not completed within the planned milestone.
Note: Confidential issues will be skipped as part of the missed label application. Please see the this issue for more information
>= 19th if the 22nd is on a Monday>= 20th if the 22nd is on a Sunday>= 21st otherwise~missed:x.y is applied, where x.y is the current milestone~Deliverable label, the ~missed-deliverable label is appliedOpen issues and merge requests planned as ~Deliverable but have a ~missed:x.y
label will have the ~missed-deliverable label applied.
Note: Confidential issues will be skipped as part of the missed label application. Please see the this issue for more information
~Deliverable
label and a ~missed:x.y label, and no ~missed-deliverable label.~missed-deliverable is applied.Issues which have a label of ~Deliverable without a milestone will have the milestone set to %Backlog.
~Deliverable without a milestone~Deliverable label is removed%BacklogIssues which have a severity label and missed the SLO target will be labeled with ~missed-SLO. The calculation for elapsed time starts from the date of the severity label was applied. This enables reporting on SLO target adherence.
~severity::1 and ~severity::2 bugs.~missed-SLO is applied.Bugs which have a severity 1 or severity 2 label without a priority label will be labeled with the equal priority label. For example, a ~severity::1 ~"type::bug" without a priority label will have ~priority::1 applied.
~"type::bug" issue with ~severity::1 or ~severity::2 without a ~priority::* label.~priority::* label of the same levelIssues or merge requests that have a label of ~"master:broken" will have labels of ~"priority::1" and ~"severity::1" applied. This ensures that requests which break master are sufficiently categorized for reporting.
~"master:broken" label.~"priority::1" and ~"severity::1" labels are applied.This automation identifies potential and popular proposals using upvotes. This helps identify feature proposals that people have indicated they would like.
~"potential proposal" or ~"popular proposal" is applied depending on the condition.GitLab values the time spent by contributors on reporting bugs. However, if a bug remains inactive for a very long period, it will qualify for auto-closure. The following is the policy for identification and auto-closure of inactive bugs.
~"severity::3" or ~"severity::4" ~"type::bug" issue is inactive for at least 12 months, it will be
identified as eligible for auto-closure. At this point, the following actions occur:
~"vintage" to indicate the issue has been inactive for a year.~"stale" to indicate that it is currently being identified for auto-closure.~"auto-closed" is applied.~"stale" is removedTier labels should be applied to issues to specify the license tier of feature. This policy prompts the Product Manager for the applied group label to add the license tier label to issues that are scheduled for the current milestone and labelled with ~direction.
The possible tier labels to be applied are:
~"GitLab Ultimate"
~direction label, in current milestoneType labels are applied to issues to increase the visibility and discoverability during team issue refinement. This policy applies to gitlab-org team member created issues and prompts the author to apply a type label to the issue within the first week.
Type labels ensure that issues are present in the group triage report and added to the correct section.
gitlab-org memberBugs have a severity label that indicates the SLO for a fix. This automated policy aims to prompt managers about bugs in their group that are approaching the SLO threshold
~"type::bug" and has a ~severity::1 or ~severity::2Issues with the ~infradev label should have a severity label, a priority label, and a milestone set. This automated policy aims to prompt managers about such issues missing one of these attributes.
~infradev and has no severity label, or no priority label, or no milestone set~"automation:infradev-missing-labels" set~"automation:infradev-missing-labels" is appliedNote:
~"automation:infradev-missing-labels" is automatically removed when a severity label, a priority label, and a milestone are set on the issue.~"automation:infradev-missing-labels" is automatically removed after two weeks, leading to a new message being posted if the Automation Conditions above are still met.
This effectively ensures that a reminder is posted on the issue every two weeks.