We enable global software organizations and teams to make great decisions with smart feedback loops by delivering speedy, reliable pipelines in a comprehensive CI platform that embodies Operations for All.
As engineers in Verify we know our customers because we are our customers, and we are constantly striving to make our platform better for everyone. We do this through iteration, dogfooding, and being involved in our open source community. We innovate, we collaborate, and we challenge assumptions to arrive at great results.
We take ownership of the things we build, with a focus on stability and availability. We do this by having a deep technical understanding of the operation and performance characteristics of our platform, and a proactive perspective to future growth.
The Verify stage is made up of 4 groups:
For of our UX Vision and Strategy, take a look at the Verify UX Strategy page.
A Job to be Done (JTBD) is a framework, or lens, for viewing products and solutions in terms of the jobs customers are trying to achieve.
Pipeline Authoring and Pipeline Execution are closely related but they also represent different stages in the cycle of a user's interaction with a pipeline. At a very high-level, this image illustrates the main focus of each group and how they can both support a better pipeline experience.
| Person | Role |
|---|---|
| Sam Beckham | Frontend Engineering Manager, Verify |
| Jose Ivan Vargas | Senior Frontend Engineer, Verify:Pipeline Execution |
| Payton Burdette | Senior Frontend Engineer, Verify:Pipeline Execution |
Each group in Verify is encouraged to define their own process based on what works best for them. Individual group processes are described on the pages below:
In the Verify team we lean in to the GitLab mission, "everyone can contribute"!
To help balance this workload out across the groups, we use the Verify candidate label.
Every issue with this label is a good candidate to be worked on by any group in the Verify stage.
This applies to both frontend and backend issues.
Prioritization is still determined by the Product Manager, such as ensuring any deliverables in the engineer's own group still take priority, but engineers are encouraged to pick work up from this board.
This helps us break down silos, balance the workload, and prevent disruptive re-allocations.
To help with prioritizing within the list of available Verify candidate issues, it's recommended to reference the issue types in the Product Priorities list, noting any severities applied on the issues as well.
Across Verify we value Transparency, we live our values of Inclusion, and we expect Efficiency by dogfooding using Issue Health Statuses and providing regular updates on active issues. Each team in the Verify Stage will define the cadence of updates and specific defintion of the statuses, but generally the expectation is a weekly update on in progress issues with the following Health Statuses:
These updates are an opportunity for the engineer to add detail to the status and are not expected to provide a justification for why something is behind or will miss a milestone. We encourage blameless problem solving and kindness at all times.
In the Verify stage, we value MR Rate as a shared performance indicator for team collaboration, iteration, customer results. The entire team is responsible for iterative scope in issues. This starts with product management creating a clear problem statement connected to user insights. UX then adds interaction specifications and acceptance criteria to then be considered and weighed by the engineering team. Teams are encouraged to iterate on scope so as to delivery the smallest thing possible.
By considering MR Rate as a measure of throughput, product management is focused on creating decomposed pieces of scope to improve the user experience. This encourages the UX and engineering teammates to provide simpler ways to solve the same problem, ultimately improving the throughput of the entire team.
Many of the groups in CI/CD have separate Frontend and Backend Engineering Managers. Given each group's ability to define their own process, groups may also consider decoupling project management functions from people management functions.
For example, a group's managers could determine that one of them will be the DRI for a given project, or the entire group. That individual would then be responsible for planning for the whole team, instead of splitting the planning responsiblity between two managers which requires extra coordination.
The following members of other functional teams are our stable counterparts:
In FY23-Q1, we're piloting async work weeks in Verify and have proposed 2021-02-13 to 2022-02-19. We're considering doing this once per quarter in the Verify stage. Some of the benefits include reducing time spent in sync meetings, allowing for more focus, which aligns with our async-first communication and our Diversity and Inclusion value to bias towards more async communication. However, this doesn't preclude us from having any meetings, and it's up to meeting facilitators to decide accordingly. Exceptions might include: high priority issues and initiatives, social calls, coffee catch-ups.
| Quarter | Async Week |
|---|---|
| FY23-Q1 | 2021-02-13 to 2022-02-19 |
TBD - we're looking to revamp this for 2022!
Every two weeks the Verify Engineering Update Newsletter is set out to an opt-in subscriber list. The purpose of the email is to share recent highlights from the Verify stage so folks will have a better idea of what is happening on other teams, and provide new opportunities for learning and collaboration.
Everyone is welcome to sign up or view previous issue on the newsletter page.
Each issue of the newsletter is planned using individual issues linked in the newsletter epic. Content is generally contributed by managers, but everyone is encouraged to contribute topics for the newsletter.
The Verify stage uses the #s_verify-updates channel in Slack to communicate various updates relevant to the group. Everyone in the stage is encouraged to contribute any sort of team, product, or process update. This channel is also a good place to celebrate sucessess and thank fellow team members.
Verify Technical Discussions is a Zoom meeting hosted monthly by the team members in the Verify stage. Everyone is invited, however paricipation from the Verify stage members is especially encouraged.
During the meeting we discuss a variety of technical aspects related to the Verify stage roadmap. Folks are also encouraged to challenges they're facing working on problems in the CI domain.
Everyone can add their points to the agenda document.
Below you can find a table with links to recordings of Verify Technical Discussions and Technical Deep Dives.
The Verify Technical Discussions are automatically recorded and added to Google Drive (internal).
| Date | Title | Recording |
|---|---|---|
| 2021-01-21 | Technical Discussions - Pipeline Editor and database storage | Recording |
| 2021-01-07 | Technical Discussions - Next iteration of CI/CD Pipeline DAG | Recording |
| 2020-12-10 | Technical Deep Dive - Observability at GitLab with demos | Recording |
| 2020-11-19 | Technical Deep Dive - Cloud Native Build Logs feature overview | Recording |
| 2020-05-08 | Technical Deep Dive - Using Prometheus with GitLab Compose Kit | Recording |
The Product Manager, Engineering Manager(s), and Designer for each group are responsible for triaging and scheduling feature proposals and bugs as part of the Weekly Triage Report. Product Managers schedule issues by assigning them to a Milestone or the Backlog. For bug triage, Engineering Managers and Product Managers work together to ensure that the correct severity and priority labels have been applied. Since Product Managers are the DRIs for prioritization, they will validate and/or apply the initial priority label to bugs. Engineering Managers are responsible for adding or updating the severity labels for bugs listed in the triage report, and to help Product Management with understanding the criticality and technical feasibility of the bug, as needed.
While SLOs for resolving bugs are tied to severities, it is up to Product Management to set priorities for the group with an appropriate target resolution time. For example, criteria such as the volume of severity::2 level bugs may make it appropriate for the Product Manager to adjust the priority accordingly to reflect the expected time to resolution.
In the Verify Stage, we aim to solve new availability, security, performance issues within the SLO of the assigned severity. These types of issues are the top priority followed by bugs and technical debt according to our severity SLO chart.
Availability and performance issues (commonly referred to as infradev) are also triaged in the Infra/Dev Triage Board.
We believe in supporting our open source community. We aim to support two main measure of success:
Each team in the Verify Stage follows roughly the same process to ensure the community is effectively supported and free to add features or fixes to the product. How we manage the Community Contribution MRs is spread across three main areas: processing the contributions, reviewing the contributions, and merging the contributions.
Code contributions to Verify typically occur in three flavors:
Contributions from both free and paid users are equally important and will follow our GitLab Contribution Guidelines. We strive to make this process as frictionless as possible between our users and the Engineering teams in Verify, especially during the reviewing of contributions.
Once a contribution has been created, the Engineering Manager assigns an engineer to manage and review with the Community Contributor. Reviewing contributions will follow the definition of done, style guidelines, and other practices of development. As the DRI of the review, the assigned Verify engineer will work with Community Contributor to resolve any outstanding items. The MR is then passed to a Maintainer with relevant domain expertise for final review prior to being merged.
Our partners are an important part of our ecosystem at GitLab. These contributions should be reviewed with the same GitLab Contribution Guidelines as community MR contributions, and aligns with the Verify contribution guidelines for working in the Verify areas of the codebase.
The Maintainer of the codebase will be the DRI of merging the contribution into the Verify product.
For issues that are refined, they are considered priority for review and merging. Refined issues are defined as issues in workflow::ready for development, with direction labels, and either have technical proposals or are weighted. Issues that are not labeled with workflow::ready for development and direction labels are considered non-refined and are lower priority for review, and therefore have longer merge SLOs. SLOs for these two types of issues are defined in the table below:
| Types of issues & users | Time Frame for Review SLO | Time frame for Merge SLO |
|---|---|---|
| All users contributing to refined issues or bugs of severity S2 or S1 | 30 days | The next release (60 days) |
| Paying users from non-refined issues or bugs of severity S3 | 60 days | Within the next 3 releases (approx one quarter or 90 days) |
| Non-paying users from non-refined issues or bugs of severity S4 | 90 days | Anything outside the next 3 releases (more than one quarter or 120 days). |
In order to prevent the inflow of Community Contributions overwhelming engineers on the team and impacting their ability to work on planned issues, there is a WIP limit of 5 assigned Community Contribution MRs per reviewing engineer. This helps limit the amount of context switching a single engineer is doing and prevents them from being overwhelmed with reviews.
