Any GitLab team-member can triage issues. Keeping the number of un-triaged issues low is essential for maintainability, and is our collective responsibility.
We have implemented automation and tooling to handle this at scale and distribute the load to each team or group.
Video introduction to triage operations, triage report, priority and severity labels.
The Quality Engineering Department ensures that every Product and Engineering group is held accountable to deliver on the SLA set forth.
Our defect SLA can be viewed at:
The Quality Engineering department employs a number of tools and automation in addition to manual intervention to help us achieve this goal. The work in this area can been seen in our department roadmap under Triage and Measure tracks of work.
There is a large amount of automation that uses stage, group, and category labels. We ask that Product Managers create an issue in triage-ops when any of the following changes occur. This issue helps ensure limited to no impact to automation and reports.
Our triage bot will automatically infer section, stage, and group labels based
on the category/feature already set on an issue or MR. This is available for open issues/MRs within the gitlab-org group.
The most important rules are:
stages.yml and the label is already set.The following logic was initially implemented in this merge request:
After the above inference is done, a section label will be added based on the stage or group label. An explanation will not be added in this step if the inferred labels contain only a section label.
Check out the list of actual use-cases to better understand what this flow means in practice.
If your issue/MR doesn't belong to a particular stage, you can remove the stage
label and add the ~"automation:devops-mapping-disable" label to prevent this
automation from happening in the future.
A triage report is an issue containing a checklist of issues requiring attention. Each task corresponds to an issue that needs labels, prioritization and/or scheduling.
An issue created by a triage report is automatically assigned to team members. Those team members are listed in group definition file, or the respective triage report policy YAML files.
To change who an issue gets assigned to, open a merge request for the above files. If the group definition file is changed, we'll need to run some scripts to update the generated files as well.
This report contains the 72 most recent unlabelled issues requiring initial triage. The goal is to ensure we achieve partial triage before the issue is picked up by a Product Manager and Engineering Manager in that area.
Sometimes the unlabelled triage report creation fails due to spam detection. When the issue is not created, a notification for failure of the verify-unlabelled job will be raised to the #triage-automations channel.
The issue can be created manually using the following command:
bundle exec gitlab-triage --debug --r ./plugins/all.rb --token PERSONAL_API_TOKEN --source projects --source-id gitlab-org/gitlab -f ./policies/stages/report/unlabelled-issues.yml
This report contains recent wider community contribution merge requests requiring initial triage. The goal is for coaches to add stage and group labels (as well as type and category labels, ideally), so that the relevant Product Manager or Engineering Manager can be pinged later on based on these labels.
This report contains the relevant bugs, feature requests, and UX debt issues that belong to a group in our DevOps stages. The goal is to achieve complete-triage by the Product Manager, Engineering Manager, UX team member in that area.
The report itself is divided into 4 main parts.
~priority::1 and ~priority::2 bugs past the target SLO.The bug sections also contains a heatmap.
An example: https://gitlab.com/gitlab-org/quality/triage-ops/issues/118
Video overview of the triage report.
There is also an optional stage policy for missing categories. If your team has enabled this, you will receive a list of up to 100 items that have the stage label but have zero appropriate category labels for that stage.
This section contains issues with the ~"feature" label without a milestone. It is divided further into issues with and without ~"customer"
Backlog or Awaiting further demand milestone.This section contains issues with the ~"bug" and ~"frontend" labels without priority and severity. It is divided further into issues with and without ~"customer"
Backlog.This section contains issues with the ~"bug" label without priority and severity. It is divided further into issues with and without ~"customer"
Backlog.This section contains bugs which has past our targeted SLO based on the severity label set. This is based on our missed SLO detection triage policy.
This report contains merge requests that belong to a group in our DevOps stages. It's composed of three sections:
Some merge requests are idle with no activity for them and are merged more than 28 days from when they are opened. This report attempts to collect them for identifying the actions we need to take, such as nudging the author, reviewer, or maintainer.
An example report: Merge requests requiring attention for group::access - 2020-11-08. Current reports can be found in the triage-reports project
Using actual links to the merge requests will update the merge request which will remove them from the future report.
This report contains feature flags that have enabled in the codebase for 2 or more releases for groups within our DevOps stages.
The DRI is responsible for reviewing these feature flags to determine if they are able to be removed entirely, or create separate issues to ensure the overdue feature flags are removed accordingly.
An example report: Feature Flags requiring attention for group::continuous integration - 2021-03-01. Current reports can be found in the triage-reports project
The feature flag triage reports are generated by the gitlab-feature-flag-alert project.
This report contains open merge requests which has been submitted by the wider
community. These merge requests would have the ~"Community contribution" label.
The report itself is divided into 2 parts. The first part contains the newest unassigned merge requests from the wider community. The second part contains merge requests that weren't updated for 28 days or more.
Reports open for more than 2 weeks with the ~"triage report" label will be closed automatically with the close old triage reports automation.
General triage automation is run to label and update issues which help with reporting and milestone transition. This is handled by triage-ops.
Open issues and merge requests that have missed the current release will be rescheduled to the next active milestone. This identifies pending work that was not completed within the planned milestone.
Note: Confidential issues will be skipped as part of the missed label application. Please see the this issue for more information
>= 19th if the 22nd is on a Monday>= 20th if the 22nd is on a Sunday>= 21st otherwise~missed:x.y is applied, where x.y is the current milestone~Deliverable label, the ~missed-deliverable label is appliedOpen issues and merge requests planned as ~Deliverable but have a ~missed:x.y
label will have the ~missed-deliverable label applied.
Note: Confidential issues will be skipped as part of the missed label application. Please see the this issue for more information
~Deliverable
label and a ~missed:x.y label, and no ~missed-deliverable label.~missed-deliverable is applied.Issues which have a label of ~Deliverable without a milestone will have the milestone set to %Backlog.
~Deliverable without a milestone~Deliverable label is removed%BacklogIssues which have a severity label and missed the SLO target will be labeled with ~missed-SLO. The calculation for elapsed time starts from the date of the severity label was applied. This enables reporting on SLO target adherence.
~severity::1 and ~severity::2 bugs.~missed-SLO is applied.Bugs which have a severity 1 or severity 2 label without a priority label will be labeled with the equal priority label. For example, a ~severity::1 ~bug without a priority label will have ~priority::1 applied.
~bug issue with ~severity::1 or ~severity::2 without a ~priority::* label.~priority::* label of the same levelWhen milestone is present on an issue but there is not an assignee. The milestone being present indicates the product team has reviewed and scheduled the issue. This encourages open source contributions for planned features.
Issues with the ~workflow::blocked, ~workflow::design, ~workflow::planning breakdown, ~workflow::refinement or ~workflow::verification labels are excluded from this rule.
~"Accepting merge request" label is applied.Issues or merge requests that have a label of ~"master:broken" will have labels of ~"priority::1" and ~"severity::1" applied. This ensures that requests which break master are sufficiently categorized for reporting.
~"master:broken" label.~"priority::1" and ~"severity::1" labels are applied.This automation identifies potential and popular proposals using upvotes. This helps identify feature proposals that people have indicated they would like.
~"potential proposal" or ~"popular proposal" is applied depending on the condition.Merge requests which have an author that is not a member of gitlab-org will have a label of ~"Community contribution" applied. This informs the GitLab community team about new community contributions.
gitlab-org with author that is not in the gitlab-org group.gitlab-com/www-gitlab-com with author that is not in the gitlab-com group.~"Community contribution" is appliedMerged merge requests with the ~"Community contribution" label and no milestone will automatically get the relevant milestone set. This helps keep the community contributions numbers accurate.
~"Community contribution" label, and no milestone.merged_at of the merge request and the start_date and due_date of the milestoneGitLab values the time spent by contributors on reporting bugs. However, if a bug remains inactive for a very long period, it will qualify for auto-closure. The following is the policy for identification and auto-closure of inactive bugs.
~"severity::3" or ~"severity::4" ~"bug" issue is inactive for at least 12 months, it will be
identified as eligible for auto-closure. At this point, the following actions occur:
~"vintage" to indicate the issue has been inactive for a year.~"stale" to indicate that it is currently being identified for auto-closure.~"auto-closed" is applied.~"stale" is removedTier labels should be applied to issues to specify the license tier of feature. This policy prompts the Product Manager for the applied group label to add the license tier label to issues that are scheduled for the current milestone and labelled with ~direction.
The possible tier labels to be applied are:
~"GitLab Ultimate"
~direction label, in current milestoneType labels are applied to issues to increase the visibility and discoverability during team issue refinement. This policy applies to gitlab-org team member created issues and prompts the author to apply a type label to the issue within the first week.
Type labels ensure that issues are present in the group triage report and added to the correct section.
gitlab-org memberBugs have a severity label that indicates the SLO for a fix. This automated policy aims to prompt managers about bugs in their group that are approaching the SLO threshold
~bug and has a ~severity::1 or ~severity::2Issues with the ~infradev label should have a severity label, a priority label, and a milestone set. This automated policy aims to prompt managers about such issues missing one of these attributes.
~infradev and has no severity label, or no priority label, or no milestone set~"automation:infradev-missing-labels" set~"automation:infradev-missing-labels" is appliedNote:
~"automation:infradev-missing-labels" is automatically removed when a severity label, a priority label, and a milestone are set on the issue.~"automation:infradev-missing-labels" is automatically removed after two weeks, leading to a new message being posted if the Automation Conditions above are still met.
This effectively ensures that a reminder is posted on the issue every two weeks.Reactive triage automation is complementary to general triage automation where realtime feedback provides an improved developer experience. This is handled by triage-serverless.
Merge requests which have an author that is not a member of gitlab-org will have a label of ~"Community contribution" applied.
Additionally, a note will be posted to thank the contributor for their contribution, and to give some information about the review process, as well as some useful resource links.
gitlab-org~"Community contribution"For issues labelled ~"availability", the minimal are enforced with the
guidelines at
https://about.gitlab.com/handbook/engineering/quality/issue-triage/#availability-prioritization
Whenever ~"backstage [DEPRECATED]" is added, it'll remove it and hint
about why it should not be added, and alternatives will be provided.
The ~"customer" label is applied when a customer associated link is applied.
The following URLs are considered customer associated links:
gitlab.zendesk.comgitlab.my.salesforce.comWhenever a subtype label is added, the corresponding type label is added. Current type labels with subtype labels are:
~"feature"~"tooling"