For an understanding of what this team is going to be working on take a look at the product vision.
The Release:Progressive Delivery is focused on all the functionality with respect to Continuous Delivery and Release Automation.
This team maps to Release devops stage.
The following people are permanent members of the Release Team:
|Chase Southard||Backend Engineering Manager, Release:Progressive Delivery|
|Shinya Maeda||Senior Backend Engineer, Release:Progressive Delivery|
|Jason Goodman||Backend Engineer, Release:Progressive Delivery|
|Etienne Baqué||Backend Engineer, Release:Progressive Delivery|
|Amy Troschinetz||Senior Backend Engineer, Release:Progressive Delivery|
|Nicole Williams||Frontend Engineering Manager, Release (CD)|
|Andrew Fontaine||Frontend Engineer, Release:Progressive Delivery|
|Angelo Gulina||Senior Frontend Engineer, Release:Progressive Delivery|
The following members of other functional teams are our stable counterparts:
|Rayana Verissimo||Senior Product Designer, Release:Release Management|
|Orit Golowinski||Senior Product Manager, Release:Progressive Delivery|
|Jackie Meshell||Senior Product Manager, Release:Release Management|
|Lorie Whitaker||Senior UX Researcher, Release and Verify & Enablement|
|Daniel Croft||Senior Manager, Engineering, Package & Release (interim)|
|Marcel Amirault||Technical Writer, Verify (Continuous Integration, Testing), Release (Progressive Delivery)|
|Suzanne Selhorn||Senior Technical Writer, Package, Release (Release Management), Verify (Runner)|
Like most GitLab backend teams, we spend a lot of time working in Rails on the main GitLab CE app, but we also do a lot of work in Go which is used heavily in GitLab Pages. Familiarity with Docker and Kubernetes is also useful on our team.
Our planning and build process is detailed on our planning page.
We use a release planning issue to plan our release-level priorities over each milestone. This issue contains individual sections for deliverables, capacity, team member vacation, and more. This issue allows the entire group to view high-level planning including what we seek to accomplish in a given release. This type of issue is the central location for information gathering and planning conversations.
Issues that contribute to the progressive delivery group of the release stage of the devops toolchain have the
~"group::Progressive Delivery" labels.
As described below, we use a successive set of discrete labels and issue boards built from those labels in a mostly linear flow that is similar to Kanban. In a kanban-like flow, issues/cards are pulled not pushed. Individuals on the team should self-assign to issues in the
Ready for Development columns. On occasion, the Engineering Manager will assign a designer, developer, or other contributor as required.
The team's primary Slack channel is
#g_progressive-delivery, and the public managers channel is
#release-stage-managers. We also support a number of feature channels for discussons or questions about a specific feature area. We no longer support issue specific channels, as they are easy to lose track of and fragment the discussion. Supported channels are:
Ahead of the near-term planning phase there is an estimation phase. See the engineering evaluation and estimation section below.
We use a Milestone Planning Board customized for each milestone to organize and plan upcoming features. This board details the various stages an issue can exist in as it is being readied for development. It's not necessary that an issue go through all of these stages, and it is allowed for issues to move back and forward through the workflow as they are iterated on. It highlights
bugs above other priority issues. This is use in concert with the async release planning issue (see below).
Issues that are being planned should include the
cicd::planning issue label so as to distinguish them from those issues which will be worked on during this milestone. We reserve the
cicd::active label as the counterpart to
cicd::planning for active work.
Once we have determined the cutline for a given milestone based on our capacity, spillover from the previous milestone, bugs, security fixes, tech debt, and UX debt, we can move those issues into
~workflow::scheduling labels and a customized
%milestone assignement filter are used to feed the scheduling board where the EM and PM connect on scheduling upcoming issues.
We use the Scheduling Board as a checkpoint to ensure that we have well-formed issues and have estimated our capacity correctly prior to labelling these issues ~"workflow::ready for development".
~workflow::ready for development and
cicd::active labels are required before they're scheduled in an iteration.
We use the Build board to track features as they make their way from idea to production. This should is the only place that developers should find new work. This board does not have a concept of a Milestone, it represents the development lifecycle and is populated with issues that have been previously evaluated. On occasion, unscheduled issues like high priority bugs, community contributions, the like will appear on this board. Here, issues move more or less linearly from Ready for Development thru to Verification.
On occasion they have to return to and earlier state like
workflow:ready for development as priorities shift,
workflow::refinement if the issue was not truly ready, or to
workflow::blocked if we are unable to proceed. As a point of clarification,
workflow::scheduling contains issues that are ready to go and that we're waiting to act until we have capacity or waiting planned for specific milestone while
workflow::refinement is about the proposal needing polishing.
Issues if over a weight of 1, will have multiple MRs. Each of these associated MRs will likely be in a different stage of development. It's important to update the issues on the build board with the latest, most relevant state. Example, if an issue has three MRs with one in review, another in development, and one ready for development, then the we should place that issue is in ~"workflow::In dev" as this is where the most activity for this issue is happening - it's the most relevant signal to the rest of the team. Because we have a one-to-many relationship between our issues and MRs, activity on the board will appear slow. Async Issue Updates and weekly sync meetings both outlined below help with any ambiguity that this system introduces.
Below is a description of each stage by label, its meaning, and any requirements for that stage.
workflow::ready for development
Before the team will accept an issue into a milestone (as marked by the the labels
~workflow::"ready for development"
+ ~cicd::active`) for work it must meet the following criteria:
~featureinclude a well stated "why" and the customer problem to solve
~buginclude, steps to reproduce
backendor both to indicate which areas will need focus.
There are a number of labels applied at any time to both issues and merge requests, but there is a priority, first being the highest:
planning priority- This label is rarely used, but we consider it to be a promise to a customer. Issues with this label will supersede all other work even those planned issues. The use of this label is usually held in reserve for issues of the utmost importance.
S1- From the perspective of bugs, a
S1has the highest priority and a short SLO. Issues with this label will supersede all other work even planned issues.
Deliverable- Issues with this label are agreed between team EM and the PM to have the highest overall normal priority and could be delivered in the given milestone. Deliverable issues not closed in a given milestone are auto forwarded via milestone cleanup workflow. Deliverable issues are often paired with
Release::P1- Similar to and often paired with
Deliverable. This is the highest priority from the perspective of the sub-department. Assuming there are no
P1/S1issues, we start with these issues first, and we focus on these over lower priority issues as much as possible.
Community contribution- The main label for the Contributor Program. We should support the contributor community by triaging and coaching issues in our area with this label. We prioritize this effort between Deliverables and lower priority issues.
Release::P2- This is the second highest priority from the perspective of the sub-department.
Release::P3- This is the third highest priority from the perspective of the sub-department.
Unscheduled- Items with this label are being worked on in this release but have not been previously scheduled by the EM and PM. Work orignating with this label is usually event driven - another team requires immediate help, a regression affecting users, or severe technical debt that is causing inefficiency in the team.
Stuff that should Just Work- This part of our incremental improvement engineering process where should allow for capacity to make improvements to the GitLab codebase or documentation leaving our work environment a better place than when we found it.
GitLab has a documented a clear Definition of Done for contributions to GitLab. We will follow that standard as our own definition of done.
We have established a Work in Progress limit (WIP) of 7 given that we currently have 5 engineers on the team (1.5 x number of engineers). This has been applied to the
workflow::in dev list on the build board. When we've hit our maximum, should instead help move WIP issues along. The EMs will reevaluate this WIP limit with each milestone and make adjustments accordingly.
Historically, our average MRs per month (not milestone) has been about 27 MRs. (see this periscope chart)
With the knowledge that things take longer than you think, we will devote about 30% of our estimated capacity for a given milestone to
Deliverable issues. The rest of the milestone capacity is taken with assisting your peers, followed by peer reviews, maintainer reviews, community contributions, lower priority items, bugs, issue estimations, and fixing stuff that should just work. The EM and PM will evaluate this capacity allocation for Deliverable issues during milestone planning and this may adjust overtime.
In order to make sure iterations are efficient, it is important to ensure issues are clearly defined and broken down before the start of an iteration.
In order to accomplish this, engineering evaluation will take place for issues that require it (small changes and bug fixes typically won’t require this analysis).
With each milestone we collect a series of upcoming issues to be review and discussed asynchronously into a meta Needs Weight Issue (example issue). These issues should have the label
workflow::planning breakdown. We use this label not only for well-defined user stories, but also functional, performance, documentation, and security issues.
See the Build phase of the Product Development Workflow.
Assigned engineers will work with Product, UX and other engineers to determine the implementation plan for a proposed feature. As the assigned engineer provides a propsed breakdown, they should also affix a weight to the issue. See below.
Once an implementation plan has been finalized, the following steps should be taken:
workflow::ready for developmentby the Engineering Manager only after consulting with the PM during scheduling. Capacity, cut line, an issue that does not meet the definition of ready are some sceanarios that could prevent this label from being applied during planning.
Issues with a weight greater than 5 should be reconsidered by the team. This is a signal that the issue is too large, too complicated, and that we should iterate on the issue proposal. Please consider proposing a new scope of the issue such that it can be split into future iterations.
As a byproduct of the engineering evaluation process, a rough estimate of the number of merge requests required to develop a feature will be produced. This measure can be used as a way to determine issue weights. These weights can be useful during the planning process to get a rough gauge of how many features can be worked on in a milestone, and to understand how much work the team can do in a milestone. This metric also aligns with the throughput metric currently measured by engineering teams.
As a general rule, we should avoid adding interdependent Merge Requests into a single milestone except under special circumstances like a
Given our dependence on availability of reviewers and ultimately maintainers, we are not always able to guarantee a perfect coordinated release of all MRs required for a given feature sometimes 1 or more MRs is blocked by review, merge conflicts, or other entanglements.
We should take advantage of Feature Flags and other tactics if we need to neutralize behavior as needed for feature rollouts.
If you have no idea where to begin the development due to lack of knowledge on the context or you're less confident on the current technical proposal, it might be better to work on Proof-of-Concept MR (PoC or Prototype) at first. This gives you more insights on the current implementation details and potential required changes.
This step removes a lot of "if"s from your assumptions. When you're making a technical proposal, often you don't have enough time to investigate the actual implementation details. In such case, you're assuming like "Probably, this feature works like …" or "Probably, this feature and that feature are connected like …", which leaves some probability in your development plan, and if it turned out a wrong assumption, you may need to rethink the whole approach, even though you're at the middle of development cycle. PoC step derisks such a turnaround by cross-checking the technical proposal with domain/performance/security experts and UX/PM.
Here is how to work on PoC step:
Technically, if you've done a PoC step, there is no need to ask additional reviews in actual MRs as long as you don't change the feature behavior. You can simply focus on general engineering review or documentation review, only. For example, improving code quality, refactoring code, writing tests, writing documents, etc.
This is an issue template for feature development. It includes some important engineering tasks to be done in general.
``` ## Technical proposal ## Feature Flag This feature is implemented behind `feature-flag-name` feature flag and disabled by default. Once we've confirmed the feature is deemed stable, we remove the feature flag in order to publish the feature as GA. ## Planned MRs ### Backend - [ ] [MR-1 title](MR link if it already exists) - [ ] [MR-2 title](MR link if it already exists) ### Frontend - [ ] [MR-1 title](MR link if it already exists) - [ ] [MR-2 title](MR link if it already exists) ### General - [ ] [Write a feature spec to test frontend and backend change altogether](MR link if it already exists) - [ ] [Remove the feature flag and update documentation](MR link if it already exists) # i.e. publish the feature ```
This is an issue template for feature evaluation. It includes some important engineering tasks to be done in general.
When you enable a feature via feature flag and expect a significant impact on user workflow or production load, you should create an issue with the following template to communicate better with the affected users or SRE.
``` ## Summary ## Feature details ## Timeline ## How to enable the feature ## How to disable the feature ## Beginning of evaluation - [ ] Announce in Slack/CompanyCall (T-minus 1 day) - [ ] Enable the feature ## End of evaluation - [ ] Announce in Slack/CompanyCall - [ ] Disable the feature ## Feedback/Metrics ### The second evaluation ### The first evaluation ```
Code reviews follow the standard process of using the reviewer roulette to choose a reviewer and a maintainer. The roulette is optional, so if a merge request contains changes that someone outside our group may not fully understand in depth, it is encouraged that a member of the Release team be chosen for the preliminary review to focus on correctly solving the problem. The intent is to leave this choice to the discretion of the engineer but raise the idea that fellow Release team members will sometimes be best able to understand the implications of the features we are implementing. The maintainer review will then be more focused on quality and code standards.
This tactic also creates an environment to ask for early review on a WIP merge request where the solution might be better refined through collaboration and also allows us to share knowledge across the team.
As part of the 12.9 Release Post retrospective, we're taking part in an experiment whereby we should try to avoid review roulette and the maintainer bottleneck. In this pilot, try to assign reviewers / maintainers only from the CI/CD Sub-department and not by reviewer roulette.
Results: We will evaluate at the end of 13.1 if this was helpful using the metric
mean time to merge. If this improves, we'll consider keeping this change for another milestone for further evaluation. If the impact on mean time to merge is detrimental, then we may consider dropping this process and reverting back to the previous roulette process.
Since we are a remote company, we utilize a Slack plugin called Geekbot to coordinate various status updates. There are currently 3 status updates configured in Geekbot, one is weekly and two are daily:
The Weekly Status Update is configured to run at noon on Fridays, and contains three questions:
What progress was made on your deliverables this week? (MRs and demos are good for this)
The goal with this question is to show off the work you did, even if it's only part of a feature. This could be a whole feature, a small part of a larger feature, an API to be used later, or even just a copy change.
What do you plan to work on next week? (think about what you'll be able to merge by the end of the week)
Think about what the next most important thing is to work on, and think about what part of that you can accomplish in one week. If your priorities aren't clear, talk to your manager.
Who will you need help from to accomplish your plan for next week? (tag specific individuals so they know ahead of time)
This helps to ensure that the others on the team know you'll need their help and will surface any issues earlier.
#g_progressive-deliverySlack channel. It has just one question:
Is there anything you could use a hand with today, or any blockers?
This check-in is optional and can be skipped if you don't need any help, don't have any blockers or have not made any progress since the last check-in.
On of our colleagues, Nick Kipling, has created a tool called TalTal to help make this process easier. The use of the TalTal tool is totally optional.
Below are some templates that you might find also useful.
Please consider the following suggestions with regard to async issue updates:
## Async Update #### Summary <!-- general updates, probably what you would have otherwise noted in standup --> #### MR Statuses <!-- List of each MR with a quick status and/or update, this information is somewhat duplicated and could potentially be left off since the MRs are listed at the top of the issue with labels identifying the workflow state, but I like that anyone reading an update doesn't have to go searching for the MRs --> * !99995 - in maintainer review, in doc review * !99998 - in dev, 50% complete, 80% confident
Complete: 80% Confidence: 90% Notes: expecting to go into review tomorrow
It takes something more than intelligence to act intelligently. Concern: ~frontend Issue status: 20% complete, 75% confident
MR statuses: !11111 - 80% complete, 99% confident - docs update - need to add one more section !21212 - 10% complete, 70% confident - api update - database migrations created, working on creating the rest of the functionality next
The optional Daily Social is configured to run each morning and posts to #g_cicd_social. It has just one question:
What was something awesome, fun, or interesting that you did yesterday outside of work?
This check-in is optional and can be skipped if you don't have anything to share.
Each week, we make time to sync with eachother on Tuesdays to share and discuss what's happening in the current week. Meetings are recorded and available for viewing later as needed.
In an effort to allow for everyone to participate in our weekly sync meetings despite our timezone challenges, we alternate our Tuesday sync meetings from 9am US Eastern Time on Week A to 4:30pm US Eastern Time on Week B.
We use this project to organize issues directly related to and affecting our group including Retrospective Issues.
Retrospective Issues are created and delivered by the GitLab Bot and from configuration located in async-retrospectives.
Contributing to the wider engineering retrospective can be helpful for other groups and the engineering organization as a whole. To include items in the wider retrospective, we vote on items from our own retrospectives, positive or negative, with some affirmation style emoji like +1, 👍, 💯 on or before the 1st of each month. After voting, the EM will add it to the agenda for that meeting.
We discuss iteration retrospectives using an asynchronous process similar to other planning processes where a well defined issue is available for the group to comment on our iteration successes, failures, and areas for improvement. Findings from that retrospective should inform our process refinements and should be captured here.