For an understanding of what this team is going to be working on take a look at the product vision.
The Verify:Runner Group is focused on all the functionality with respect to Runner.
This team maps to Verify devops stage.
For an understanding of what our category direction is, take a look at the direction page.
We measure the value we contribute by using Performance Indicators (PI), which we define and use to track progress. As the GitLab Runner is the engine that enables GitLab continuous integration (CI), it is essential to analyze monthly active user metrics for CI when evaluating R&D investment prioritization decisions for the Runner. We have also defined additional performance indicators specific to the Runner to evaluate R & D investment prioritization decisions.
As defined on the growth page, AARRR stands for Acquisition, Activation, Retention, Revenue, and Referral. This framework represents the customer journey, and the various means a product manager may apply a North Star metric (performance indicator) to drive a desired behavior in the funnel. As discussed above, since Runner is the engine that drives GitLab CI, at this time, we will leverage the same usage funnel definitions for Verify:CI.
When we launch the macOS Runners as a product, we will add a section here that defines the AARRR metrics for that offer. The rationale for that is that we are planning for a separate landing page for the macOS Runners. As such, we can measure page views and click-throughs to call to actions.
Example AARRR metrics that we have in mind for that offer:
The following people are permanent members of the Verify:Runner group:
| Person | Role |
|---|---|
| Elliot Rushton | Backend Engineering Manager, Verify:Runner |
| Tomasz Maczukin | Senior Backend Engineer, Verify:Runner |
| Steve Azzopardi | Backend Engineer, Verify:Runner |
| Georgi Georgiev | Senior Backend Engineer, Verify:Runner |
| Pedro Pombeiro | Senior Backend Engineer, Verify:Runner |
| Arran Walker | Senior Backend Engineer, Verify:Runner |
| Adrien Kohlbecker | Senior Backend Engineer, Verify:Runner |
To find out who our stable counterparts are, look at the runner product categtory
As a team we maintain several projects. The https://gitlab.com/gitlab-com/runner-maintainers group is added to each project with maintainer permission. We also try to align tools and versions used across them.
We spend a lot of time working in Go which is the language that GitLab Runner is written in. Familiarity with Docker and Kubernetes is also useful on our team.
We follow the same process outlined here. Any engineeer inside of the organization is welcome to become a maintainer of a project owned by the Runner team. The first step would be to become a trainee maintainer.
To start the maintainer training process, please create an issue in the Runner project's issue tracker using the Release Maintainer Trainee template.
After the training is finished, the new maintainer will be added to the https://gitlab.com/gitlab-com/runner-maintainers group, which will set proper permission across the projects.
editor access to the group-verify project in GCPdeveloper to the gitlab-com/runner-group group on GitLab.comteam.yml has the new member as a reviewer of gitlab-org/gitlab-runner and gitlab-org/ci-cd/custom-executor-drivers/autoscalerRunner Group Daily Standup and Runner Weekly RetroVerify 1password vault (requires creating an access request).We try to use the Kanban process for planning/development. Here is the single source of truth about how we work. Any future changes should be reflected here. If you feel like that process is not ideal in certain aspects or doesn't achieve a goal, you are more than welcome to open a merge request and suggest a change.
We use the following board for planning, milestone tracking.
~workflow::start: This is the backlog for the product manager, they
are responsible for this column. Issues should be vertically stacked,
the top one has the highest priority. At this stage the issue is not
very fleshed out, and we still need to understand the problem. When an
issue is in this column it means that the product manager is aware of
the issue and will start working on it soon. This column is limited to
10.~workflow::design: At this point, issues that impact the GitLab UI have a clear problem and are going through the design process, where the UX Definition of Done (DoD) is applied to any issues. In order to progress to ~workflow::planning breakdown, the UX DoD must be complete.~workflow::planning breakdown: This is where the engineering team
will work with the product manager to make sure we have a good
proposal for the issue at hand. Each engineer should spend 1-2 hours each
week to discuss the issue async with the community and product manager
to figure out a proposal. Particular attention should be given to thinking
about how the team can build the feature in an iterative way - what's the
smallest change we could make to get some version of the feature into the
hands of a customer in a single release?
Sometimes it's helpful to attach a milestone to the issue that is in this
column, so that an engineer can focus time doing PoC and research spikes for
that specific release.. It's perfectly OK for an issue to be closed from this
column if we decided to split the issue into multiple ones. When both the
product manager and engineering team are happy with the proposal they should
move the issue to the next column, ~workflow::ready for development. The
people assigned to this issue are most likely engineers and are responsible
for getting this issue to the next stage.
~workflow::ready for development: At this stage, we should have a
good idea of what the issue requires, and how much work it is. When an
engineer is out of tasks to work on and has no merge requests to
review or any issues that are ~workflow::In dev they should pick the
one on top, assign it to them and move it to the next column. In this
column, each issue should have a milestone attached to it to indicate
to the customer in which release it will be done in.~workflow::In dev: Here is where the engineer starts working on the
issue. If the issue is not clear on what needs to be achieved, they
should discuss it with the team to see if it needs to go back to
previous stages. This column is limited to 3 issues per engineer in
the team. If they have more issues assigned to them we need to
reevaluate the workload of the engineer because there is most likely a
lot of context switching which is not effective.~workflow::In review: When the issue has either 1 merge request or
multiple merge requests ready for review it should be moved to the
~workflow::In review column. There is a limit of 20 issues that
should be in review. If we are at limit engineers should not pick up
new work but see if they can help out in the review process.~workflow::blocked: There are multiple reasons why an issue can be
blocked. If the issue has a milestone attached to it, the issue
blocking it should have the same milestone or 1 earlier, and have a
higher priority. We should also mark the issue as blocked using the
related issue
feature.~workflow::planning breakdown or ~workflow::In review. If not pick
a new issue from ~workflow::ready for development.Tips:
/label
quick action to keep the same priority stack.As discussed above each issue inside of ~workflow::ready for
development should have a milestone attached to it. If stack-ranked
properly, the issues on top should be in the current milestone, and then
the upcoming milestone. The product manager will take the top 5 issues
from the ~workflow::ready for development column that is intended
for the upcoming milestone and use these issues as kickoff
issues.
Each week the EM and PM on the team get together and select 1-3 issues that are our next highest priority to be broken down. On a rough rotation, the EM will then assign those issues to individual engineers. Over the following 1-2 weeks the assigned engineers will be responsible for coming up with an implementation plan for the issue - exploring any unknowns, edge cases, or compatibility challenges that may come up and proposing how the issue can be broken down into smaller, more reasonable to handle tasks. During this time frame the other engineers on the team will also be asked to become familiar with the issue. On the team call that follows, the assigned engineers will lead a discussion about the considerations and challenges they uncovered and the implementation plan they proposed. After the discussion the assigned engineer will capture any notes from the conversation and the implementation plan in the issue itself, move the issue into ~workflow::ready for development, and unassign themselves from the issue so that it can be picked up in the priority based approach described by the Kanban workflow above.
In general, technical debt, backstage work, or other classifications of development work that don't directly contribute to a users experience with the runner are handled the same way as features or bugs and covered by the above Kanban style process. The one exception is that for each engineer on the team, they can only have 1 technical debt issue in flight at a time. This means that if they start working on a technical debt type issue they cannot start another one until the first one is merged. In the event that an engineer has more than one technical debt item in flight, they should choose which one to keep working on and move the others to the "in development" or "ready for review" columns depending on their status. The intent of this limitation is to constrain the number of technical debt issues that are in review at any given time to help ensure we always have most of our capacity available to review and iterate on features or bugs.
The team has a monthly retrospective meeting on the first Tuesday of the month. The agenda can be found here (internal link).
Each quarter we have an error budget of how many regression the release can cause.
Following the point system, each quarter we have we have 100 points. Each type of regression has a priority/severity, which every priority has a certain point associated to it, the higher the priority the more points:
~priority::1: 70~priority::2: 30~priority::3: 15~priority::4: 5Every beginning of the quarter the error budget is set to 100 again.
Each regression should have a retrospective item with corrective actions
to prevent any similar regression from happening again. These issues
should be labeled with ~"corrective action".
While Google's original paper advocates for stopping all feature work and focusing the team only on reliability improvements for the quarter, we have taken a more minimal first step: exceeding our error budget will only be a callout for our Product Manager to immediately prioritise corrective actions identified during the retrospective. Other feature work by team members not assigned to those corrective actions can proceed normally.
Below is the history of each quarter, and should be filled with the following template:
- Regressions:
- Issue
- Retrospective Issue
- Corrective Action Issues
- Error Budget: -X
- Final Error budget: 30
This was inspired by the Google Error budget. It's used to balance reliability and new features.
When a community contribution is abandoned by the original author, it's up to us to get it across the finish line, the list below outlines the process.
~"coach will finish" label to the merge request, and close
the merge request.~"Community contribution" label to it to make it clear that
it is/was being worked on by a community member.Issues worked on by the Runner group a group label of ~group::runner. Issues that contribute to the verify stage of the devops toolchain have the ~devops::verify label.
GitLab.com: @gitlab-com/runner-group
Slack: #g_runner
Our code review process follows the general process where you choose a reviewer (usually not a maintainer) and then send it over to a maintainer for the final review.
Current maintainers are members of the runner-maintainers group.
Current reviewers are members of the runner-group group.
The gitlab-runner codebase, which is the primary codebase that the runner group works in, follows a 7th-of-the-month feature freeze. This is documented with the codebase here and is important to be aware of as it drastically differs from most of the rest of the gitlab release timing.
See dedicated page.
