The Verify:Continuous Integration Group is focused on all the functionality with respect to Continuous Integration.
This team maps to Verify devops stage.
We measure the value we contribute by using Performance Indicators (PI), which we define and use to track progress. The current PI for the Continuous Integration group is the
number of unique users who trigger ci_pipelines. For more details, please check out the Product Team Performance Indicators. To view the latest Verify stage ci_pipeline data see our Sisense Dashboard.
Based on the AARRR framework (Acquisition, Activation, Retention, Revenue, Referral), this funnel represents the customer journey in using GitLab CI. Each state in the funnel is defined with a metric to measure behavior. Product managers can focus on any of the various states in the funnel to prioritize features that drive a desired action.
The following people are permanent members of the Verify:Continuous Integration group:
|Cheryl Li||Backend Engineering Manager, Verify:Continuous Integration & Verify:Pipeline Authoring|
|Matija Čupić||Backend Engineer, Verify:Continuous Integration|
|Fabio Pitino||Senior Backend Engineer, Verify:Continuous Integration|
|Allison Browne||Backend Engineer, Verify:Continuous Integration|
|Sam Beckham||Frontend Engineering Manager, Verify|
|Jose Ivan Vargas||Senior Frontend Engineer, Verify:Continuous Integration|
|Payton Burdette||Senior Frontend Engineer, Verify:Continuous Integration|
The following members of other functional teams are our stable counterparts:
|Thao Yeager||Senior Product Manager, Verify:Continuous Integration|
|Veethika Mishra||Senior Product Designer, Verify:Continuous Integration|
|Marcel Amirault||Technical Writer, Verify (Continuous Integration, Pipeline Authoring, Testing), Growth (Retention)|
|Suzanne Selhorn||Senior Technical Writer, Package, Verify (Runner), Growth (Expansion)|
|Grzegorz Bizon||Staff Backend Engineer, Verify:Continuous Integration|
|Darby Frey||Senior Manager, Engineering, Verify|
|Tiffany Rea||Software Engineer in Test, Verify:Continuous Integration|
|Lorie Whitaker||Staff UX Researcher, Verify, Package, and Release|
Like most GitLab backened teams we spend a lot of time working in Rails on the main GitLab CE app, but we also do a lot of work in Go which is the language that GitLab Runner is written in. Familiarity with Docker and Kubernetes is also useful on our team.
For those new to the team, these links may be helpful in learning more about the product and technology.
Issues are refined and weighted prior to scheduling them to an upcoming milestone. We use
candidate:: scoped labels to help with planning work in future iterations. The additional label allows us to filter on the issues we are planning, allowing Product, Engineering and UX to start async issue refinement on the 4th. Weighting also helps with capacity planning with respect to how issues are scheduled in future milestones.
We create 2 issues as part of our Planning process:
Both issues currently rely on the
candidate:: scoped label to determine what issues are to be investigated for the upcoming milestone(s).
We add a
Weight to issues as a way to estimate the effort needed to complete an issue. We factor in complexity and any additional coordination needed to work on an issue. We weight issues based on complexity, following the fibonacci sequence:
|1: Trivial||The problem is very well understood, no extra investigation is required, the exact solution is already known and just needs to be implemented, no surprises are expected, and no coordination with other teams or people is required.
Examples are documentation updates, simple regressions, and other bugs that have already been investigated and discussed and can be fixed with a few lines of code, or technical debt that we know exactly how to address, but just haven't found time for yet.
|2: Small||The problem is well understood and a solution is outlined, but a little bit of extra investigation will probably still be required to realize the solution. Few surprises are expected, if any, and no coordination with other teams or people is required.
Examples are simple features, like a new API endpoint to expose existing data or functionality, or regular bugs or performance issues where some investigation has already taken place.
|3: Medium||Features that are well understood and relatively straightforward. A solution will be outlined, and most edge cases will be considered, but some extra investigation will be required to realize the solution. Some surprises are expected, and coordination with other teams or people may be required.
Bugs that are relatively poorly understood and may not yet have a suggested solution. Significant investigation will definitely be required, but the expectation is that once the problem is found, a solution should be relatively straightforward.
Examples are regular features, potentially with a backend and frontend component, or most bugs or performance issues.
|5: Large||Features that are well understood, but known to be hard. A solution will be outlined, and major edge cases will be considered, but extra investigation will definitely be required to realize the solution. Many surprises are expected, and coordination with other teams or people is likely required.
Bugs that are very poorly understood, and will not have a suggested solution. Significant investigation will be required, and once the problem is found, a solution may not be straightforward.
Examples are large features with a backend and frontend component, or bugs or performance issues that have seen some initial investigation but have not yet been reproduced or otherwise "figured out".
We typically do not weigh an issue more than a 5, as we try to split up issues greater than 5.
We follow the same weighting system that the
Create stage uses. see the handbook page of
Create:Source Code group as an example.
To encourage more transparency and collaboration amongst the team and additionally align on the Release Posts we publish at the end of each milestone, we will be creating a separate issue to highlight a Feature flag roll out plan for each feature being released starting in 13.2, based on the issue template for feature flag roll outs. The engineer who implements the feature will be responsible for creating this separate issue to highlight the details of when and how the feature flag will be toggled, and subsquently link this issue to their feature issue. The product manager will tag this issue as a blocker to their release post, so that everyone is aligned on the release plan of the feature.
We use the Continuous Integration Workflow issue board to track what we work on in the current milestone.
Development moves through workflow states in the following order:
workflow::ready for development
Each member of the team can choose which issues to work on during a milestone by assigning the issue to themselves. When the milestone is well underway and we find ourselves looking for work, we default to working right to left on the issue board by pulling issues in the right-most column. If there is an issue that a team member can help with on the board, they should do so instead of starting new work. This includes conducting code review on issues that the team member may not be assigned to, if they feel that they can add value and help move the issue along to completion.
Specifically, this means our work is prioritized in the following order:
workflow::in devif applicable
workflow::ready for developmentfor development column
The goal of this process is to reduce the amount of work in progress (WIP) at any given time. Reducing WIP forces us to "Start less, finish more", and it also reduces cycle time. Engineers should keep in mind that the DRI for a merge request is the author(s), to reflect the importance of teamwork without diluting the notion that having a DRI is encouraged by our values.
For issues being implemented in the current milestone, we use the Issue Health Status feature to designate the high level status of the issue. This issue health status is updated by the directly responsible individual (DRI) as soon as they recognize the state of the issue has changed.
The following are definitions of the health status options:
Spikes are time-boxed investigations typically performed in agile software development. We create Spike issues when there is uncertainty on how to proceed on a feature from a technical perspective before a feature is developed.
The Continuous Integration group supports the product marketing categories described below:
||Issues||MRs||Issues related to CI build artifacts. Formerly
||Issues||MRs||Relates to functionality surrounding pre-defined and user-defined variables available in the Build environment. Formerly
||Issues||MRs||All issues and MRs related to how we count continuous integration minutes and calculate usage. Formely
||Issues||MRs||Issues related to the execution of pipeline jobs, including DAG, child pipelines, and matrix build jobs|
||Issues||MRs||Issues related to
||Issues||MRs||Issues related to API endpoints for CI features.|
||Issues||MRs||Issues related to the notifications related to CI features.|
||Issues||MRs||Issues related to CI functionality within the Merge Request.|
||Issues||MRs||Issues related to CI statistics and dashboards.|
||Issues||MRs||Any issues and merge requests related to CI/CD core domain, either as changes to be made or as observable side effects.|
||Issues||Issues that are helpful for someone onboarding as a new team member.|
||Issues||Issues that are good for first time community contributors, and could similarly be labeled for
To create a high-quality product that is functional and useful – Engineering, PM and Product Designer need to work closely together, combine methodologies, and often connect throughout the product development.
Product Designers play a critical role in the product development of user-facing issues. They collaborate with the Engineering and the Product Manager to design the user experience for the features. Once the design solution is proposed, agreed and validated, the Engineering DRI is assigned to implement that design and functionality during the milestone for which the issue is planned.
Product Designer, PM, and Engineering use
workflow::design to discuss possible complexities, challenges, and uncover blockers around the proposed solution. To avoid blocking reviews later in the product development flow, the Product Designer, PM, and Engineering should work collaboratively throughout the feature design and development process to check-in often, so that the UX approval on merge requests is not required.
Our process of planning and development relies heavily on overcommunication rather than any approval gates or automated notification mechanisms. We adhere to the proactive mindset and responsibility of everyone involved to make sure every step in the process is as transparent as it can be.
For both planning and building this means direct, cross-functional, and other relevant stakeholders are included early into the process. This makes sure everyone is able to contribute to the best of their capabilities at the right time in the process. This can include, but is not limited to, GitLab objects, Slack, meetings, and daily standups.
Some practical examples of this are:
Note: A good practice when only wanting to inform rather than requesting a direct action from the mentioned stakeholders is to put
FYI directly following the @mention handle.
We suggest using the below steps to reach the best results in the shortest time:
We aim to design broadly for an epic or full feature at least one milestone ahead of time and then break the big solution into smaller issues to pick up in the next milestones. Suppose working one milestone ahead to design the big solution is not possible. In that case, Engineering and Product Designer will define the first most technically feasible and smallest feature set (MVC) to satisfy early customers that will be implemented in the same milestone.
UX should not be seen as a "gate" for reviewing Merge Requests. To avoid blocking reviews and speed up the response time, Product Designer and the Engineering DRI should work collaboratively throughout the feature development process and check-in often, so the UX approval at the stage of reviewing merge request is not required.
Tips to avoid blocking reviews:
The Engineering DRI works with the Product Designer throughout the
workflow:in dev phase to uncover possible problems with the solution early enough that exhibit unexpected behaviour to what was originally agreed upon. If there are changes to be added that weren't agreed upon in the initial issue - a followup issue should be made and the Engineering DRI should work with the Product Manager to schedule that issue in a following milestone. This allows us to focus on cleanup over signoff, iterate quickly on issues with a low level of shame, and still make sure we accomplish what we've agreed upon. We should be careful not to hold off on completing these followup issues so that we don't build up a significant amount of UX debt issues.
If we find that solutions are consistently not matching the agreed upon design, we will hold a retrospective with the DRI, designer, and product manager to discuss where the gaps in communication are so that we can improve. It may be necessary to begin requiring a UX approval for merge requests on certain issues to help the Engineering DRI meet the requirements.
We track our technical debt using the following CI Technical Debt issue board, where we track issues in the planning phase.
This board has 2 main sections:
workflow::planning breakdownwe find issues that are currently being groomed.
workflow::ready for developmentwe find issues that clearly defined and a weight has been assigned.
severity::4labels to classify the impact of the specific tech debt item. We use the list below as a guideline to grade the impact.
severity::1Blocking or Critical
Note that a multiple factors can exist at once. In that case use your judgment to either bump the impact score or lower it. For example:
severity::2bugs. Then choose
The team leverages a monthly async retrospective process as a way to celebrate success and look for ways to improve. The process for these retrospectives aligns with the automated retrospective process used by many teams at GitLab. The process is defined here: https://gitlab.com/gitlab-org/async-retrospectives#how-it-works.
A new retrospective issue is created on the 27th of each month, and remains open until the 26th of the following month. Team members are encouraged to add comments to that issue throughout the month as things arise as a way to capture topics as they happen. The current issue can be found in https://gitlab.com/gl-retrospectives/verify/-/issues.
On the 16th of each month a summary of the milestone will be added to the issue and the team will be notified to add any additional comments to the issue.
As comments are added to the issue, team members are encourage to upvote any comments they feel are important to callout, and to add any additional discussion points as comments to the original thread.
Around the 26th of the month, or after the discussions have wrapped up the backend engineering manager will summarize the retrospective and create issues for any follow up action items that need to be addressed. They will also redact any personal information, customer names, or any other notes the team would like to keep private. Once that has been completed the issue will be made non-confidential and closed.
The team is globally distributed and separated by many timezones. This presents some challenges in how we communicate since our work days only overlap by a couple hours. We have decided as a team to embrace asynchronous communication because scheduling meetings is difficult to coordinate. We meet as a team one day per week, on Wednesdays for a quick team meeting and 1-on-1s.
Daily standup updates are posted to
Feel free to ask us questions directly in this Slack channel and someone will likely get back to you within 24 hours (by the next business day). We will use following emojis to respond to the posted question accordingly:
:eyes:to indicate that one of us has seen it
:white_check_mark:to indicate that the question has been answered
The verify stage has a separate Slack channel under
#s_verify, which encompasses the other teams of Verify.
Most spontaneous team communiation happens in issues and MRs. Our issues have a group label of
~"group::continuous integration". You can also tag a team member with
@mention in the issue if you have someone specific to ask.
If you need to call the attention of the entire group, you can tag
Welcome to the team! Whether you're transferring internally or joining GitLab as a new hire, you'll be assigned an onboarding buddy. If you are an internal transfer, your manager will appoint you an onboarding buddy so you can have someone to work with as you're getting familiarized with our codebase, our tech stack and the team.
Read over this page as a starting point, and feel free to set up regular sync or async conversations with your buddy. It's also recommended to schedule a few coffee chats to meet some members of our team. Your manager may optionally create an onboarding checklist for you to go through if it's helpful.
In the our list of issues, there are also ones with an
~onboarding label that are smaller issues to onboard onto the product. If there aren't any in the current Workflow issue board to work on, reach out to your EM/PM to see which issues you can work on as part of your onboarding period.