Verify:Continuous Integration Group

1. You are here:
2. Engineering
3. Development Department
4. Ops Sub-department
5. Verify Stage
6. Verify:Continuous Integration Group

Maintained by:

On this page

• Vision
• Mission
• Performance Indicator
  • Usage Funnel
  • Core domain
• Team Members
  • Backend
  • Frontend
• Stable Counterparts
• Technologies
• Useful Links
  • Getting Started
    • Engineering Overview
• How We Work
  • Planning
    • How Engineering Refines Issues
      • Checklist for Refining Issues
      • Steps for Refining and Weighting Issues
    • Weighting Issues
  • Release Plans
  • Workflow
    • "What do I work on next?"
    • Issue Health Status
  • Spikes
    • Guidelines
    • Category Labels
    • Feature Labels
    • Other Notable Labels
  • Collaboration with UX and Engineering
    • Inclusive Development
    • Breaking down issues for iteration
    • Reviewing Merge Requests
    • Dealing with Community Merge Requests
    • Aligning on feature development
    • Avoiding crunch times between UX and Engineering
  • Technical debt
  • Retrospectives
  • Team Communication
    • How to work with us
      • Slack
      • GitLab Issues and MRs
• Developer Onboarding

Vision

For an understanding of what this team is going to be working on take a look at the product vision and the Category direction of Continuous Integration

Mission

The Verify:Continuous Integration Group is focused on all the functionality with respect to Continuous Integration. A key focus for the CI group is delivering features that achieve the outcome we track in our performance indicator.

This team maps to Verify devops stage.

Performance Indicator

We measure the value we contribute by using Performance Indicators (PI), which we define and use to track progress. The current PI for the Continuous Integration group is the number of unique users who trigger ci_pipelines. For more details, please check out the Product Team Performance Indicators. To view the latest Verify stage ci_pipeline data see our Sisense Dashboard.

Usage Funnel

Based on the AARRR framework (Acquisition, Activation, Retention, Revenue, Referral), this funnel represents the customer journey in using GitLab CI. Each state in the funnel is defined with a metric to measure behavior. Product managers can focus on any of the various states in the funnel to prioritize features that drive a desired action.

Core domain

• Pipeline processing: processes responsible for transitions of pipelines, stages and jobs.
• Rails-Runner communication: jobs queuing, API endpoints and their underlying functionalities related to operations performed by and for Runners.
• Job artifacts: storage and management of artifacts is the gateway for many CI/CD features.

Not included in CI group's domain:

• Secrets Management, see the direction page
• Pipeline Authoring, see the direction page
• Compliance in Pipelines, see the direction page

Team Members

The following people are permanent members of the Verify:Continuous Integration group:

Backend

Frontend

Stable Counterparts

The following members of other functional teams are our stable counterparts:

Technologies

Like most GitLab backened teams we spend a lot of time working in Rails on the main GitLab CE app, but we also do a lot of work in Go which is the language that GitLab Runner is written in. Familiarity with Docker and Kubernetes is also useful on our team.

Useful Links

• Issue Tracker: ~group::continuous_integration
• Slack Channel: #g_ci
• GitLab Unfiltered: Continuous Integration group

Getting Started

For those new to the team, these links may be helpful in learning more about the product and technology.

• CI/CD Development Documentation

Engineering Overview

• CI Backend Architectural Walkthrough - May 2020
• Frontend CI product / codebase overview - June 2020

How We Work

Planning

Issues are refined and weighted prior to scheduling them to an upcoming milestone. We use candidate:: scoped labels to help with planning work in future iterations. The additional label allows us to filter on the issues we are planning, allowing Product, Engineering and UX to start async issue refinement on the 4th. Weighting also helps with capacity planning with respect to how issues are scheduled in future milestones.

We create 2 issues as part of our Planning process:

1. Planning issue - Product is the DRI in prioritizing work, with input from Engineering, UX and Technical Writers. This issue is used to discuss scheduling and team capacity. We use an issue list filtered on the group and candidate labels.
2. Needs Weight issue - Engineering is the DRI in ensuring issues are refined asynchronously and weighted accordingly, given what is planned for the milestone. This helps Product understand how much can be scheduled in the future milestone(s), once the complexity and effort of the issues are better understood. For example, issues may need to be broken down if they are too large for a milestone, or further research may be necessary, so subsequent issues or epics can be created.

Both issues currently rely on the candidate:: scoped label to determine what issues are to be investigated for the upcoming milestone(s).

How Engineering Refines Issues

side note: we prefer using Refining over Grooming

Engineers are expected to allocate approximately 4 hours each milestone to refine and weight issues assigned to them. The purpose of refining an issue is to ensure the problem statement is clear enough to provide a rough effort sizing estimate; the intention is not to provide solution validation during refinement.

Engineering uses the following handbook guidance for determining weights. If any issue needs any additional ~frontend ~backend ~Quality ~UX ~documentation reviews, they are assigned to the respective individual(s).

Checklist for Refining Issues

1. Does the issue have a problem statement in the description?
2. Does the issue have the expected behaviour described well enough for anyone to understand?
3. Does the issue explicitly define who the stakeholders are (e.g. BE, FE, PM, UX and/or Tech Writer)?
4. Does the issue have a proposal in the description? If so:
   1. Does the proposal address the problem statement?
   2. Are there any unintended side effects of the implementation?
5. Does the issue have proper labeling matching the job to be done? (e.g. bug, feature, performance)

Any one on the team can contribute to answering the questions in this checklist, but the final decisions are up to the PM and EMs.

Steps for Refining and Weighting Issues

Engineers will:

1. Go through the checklist above for refining issues assigned to them
2. Add a weight based on the definitions
3. Update the ~workflow:: label to the appropriate status, e.g.
   • ~"workflow::design" if further design refinement is needed, and let the designer know
   • ~"workflow::ready for development" when refinement is completed and a weight has been applied, signaling that it's ready for implementation and the issue can be scheduled accordingly
   • ~"workflow::planning breakdown" if more investigation and research is needed, the status does not move, and the PM and EMs should be pinged
4. Unassign themselves from the issue when they are done refining and weighting the issue

Weighting Issues

We add a Weight to issues as a way to estimate the effort needed to complete an issue. We factor in complexity and any additional coordination needed to work on an issue. We weight issues based on complexity, following the fibonacci sequence:

The maximum weighted value for an issue is a 5, and may exceed one milestone to complete given additional dependencies and/or complexity. Consider whether an issue weighted with a 5 can be broken down into smaller iterations.

If an issue requires a feature flag rollout plan, consider increasing the weight by 1 or 2, according to the effort involved in rolling out the feature flag and monitoring the new behavior.

Release Plans

To encourage more transparency and collaboration amongst the team and additionally align on the Release Posts we publish at the end of each milestone, we will be creating a separate issue to highlight a Feature flag roll out plan for each feature being released starting in 13.2, based on the issue template for feature flag roll outs. The engineer who implements the feature will be responsible for creating this separate issue to highlight the details of when and how the feature flag will be toggled, and subsquently link this issue to their feature issue. The product manager will tag this issue as a blocker to their release post, so that everyone is aligned on the release plan of the feature.

Workflow

We use the Continuous Integration Workflow issue board to track what we work on in the current milestone.

Development moves through workflow states in the following order:

1. workflow::design (if applicable)
2. workflow::planning breakdown
3. workflow::ready for development
4. workflow::in dev
5. workflow::blocked (as necessary)
6. workflow::in review
7. workflow::verification
8. workflow::production
9. Closed

workflow::planning breakdown is driven by Product, but is a collaborative effort between Product, UX and Engineering. The steps for planning breakdown typically consists of:

• Product defining or clarifying the problem statement. Product and UX will collaborate on problem validation as needed
• UX providing designs (as necessary)
• Engineering clarifying the issue description as stated, and refines and weights the issue once Product and UX have provided enough details to do so.

At any point, if an issue becomes blocked, it would be in the workflow::blocked status. If there is a blocking issue, it needs to be added to the issue description or linked to the issue with a 'blocked by' relationship.

workflow::ready for development means that an issue has been sufficiently refined and weighted by Engineering, upon request by Product and UX

"What do I work on next?"

Each member of the team can choose which issues to work on during a milestone by assigning the issue to themselves. When the milestone is well underway and we find ourselves looking for work, we default to working right to left on the issue board by pulling issues in the right-most column. If there is an issue that a team member can help with on the board, they should do so instead of starting new work. This includes conducting code review on issues that the team member may not be assigned to, if they feel that they can add value and help move the issue along to completion.

Specifically, this means our work is prioritized in the following order:

• Any verification on code that is in ~workflow::verification or workflow::production
• Conducting code reviews on issues that are workflow::in review
• Unblocking anyone in workflow::blocked or workflow::in dev if applicable
• Then, lastly, picking from the top of the workflow::ready for development for development column

The goal of this process is to reduce the amount of work in progress (WIP) at any given time. Reducing WIP forces us to "Start less, finish more", and it also reduces cycle time. Engineers should keep in mind that the DRI for a merge request is the author(s), to reflect the importance of teamwork without diluting the notion that having a DRI is encouraged by our values.

Issue Health Status

For issues being implemented in the current milestone, we use the Issue Health Status feature to designate the high level status of the issue. This issue health status is updated by the directly responsible individual (DRI) as soon as they recognize the state of the issue has changed.

The following are definitions of the health status options:

• On Track - The issue has no current blockers, and is on schedule to be completed in the current milestone.
• Needs Attention - The issue is still likely to be completed in the current milestone, but there are setbacks or time constraints that could cause the issue to miss the deadline.
• At Risk - The issue has a high likelyhood of missing the current milestone deadline.

Spikes

Spikes are time-boxed investigations typically performed in agile software development. We create Spike issues when there is uncertainty on how to proceed on a feature from a technical perspective before a feature is developed.

Guidelines

• Spikes are typically marked as deliverable to ensure clear ownership from engineers
• Spikes are typically time-boxed to a short duration, sometimes one week, never more than one milestone.
• We should try to schedule Spikes at the beginning of an iteration so that the investigation will conclude before the planning of the following iteration.
• Limit no more than 2 Spikes per milestone per group.
• Typically multiple team members collaborate on a spike. We want to ensure we get multiple different viewpoints as well as keep the research focused and efficient.
• Engineers should consider edge cases in spikes but should not seek to solve every intricacy of every edge case

Category Labels

The Continuous Integration group supports the product marketing categories described below:

Feature Labels

Other Notable Labels

Collaboration with UX and Engineering

To create a high-quality product that is functional and useful – Engineering, PM and Product Designer need to work closely together, combine methodologies, and often connect throughout the product development. Product Management and Product Designers aim to work 3 months in advance of Engineering proposals to ensure the problem definition and solution has been adequately validated prior to building.

Product Designers play a critical role in the product development of user-facing issues. They collaborate with the Engineering and the Product Manager to design the user experience for the features. Once the design solution is proposed, agreed and validated, the Engineering DRI is assigned to implement that design and functionality during the milestone for which the issue is planned.

Product Designer, PM, and Engineering use workflow::design to discuss possible complexities, challenges, and uncover blockers around the proposed solution. To avoid blocking reviews later in the product development flow, the Product Designer, PM, and Engineering should work collaboratively throughout the feature design and development process to check-in often, so that the UX approval on merge requests is not required.

Inclusive Development

Our process of planning and development relies heavily on overcommunication rather than any approval gates or automated notification mechanisms. We adhere to the proactive mindset and responsibility of everyone involved to make sure every step in the process is as transparent as it can be.

For both planning and building this means direct, cross-functional, and other relevant stakeholders are included early into the process. This makes sure everyone is able to contribute to the best of their capabilities at the right time in the process. This can include, but is not limited to, GitLab objects, Slack, meetings, and daily standups.

Some practical examples of this are:

• When you open up or begin working on an epic, issue, or merge request consider if all stakeholders are aware of this or should be updated. If unsure, error on the side of updating rather than keeping silent.
• When making significant progress make sure this is seen by the relevant stakeholders even if you don't require immediate feedback by mentioning them rather than relying on automated email notifications. Make their involvement explicit.

Note: A good practice when only wanting to inform rather than requesting a direct action from the mentioned stakeholders is to put FYI directly following the @mention handle.

Breaking down issues for iteration

We suggest using the below steps to reach the best results in the shortest time:

• Once, through user research, we have identified a user need and have generated the big idea to meet that need, Product Designer drives the process of moving that idea into a solution prototype involving PM and Engineering Team early and often.
  • The UX Definition of Done can be applied in order to better break down design work and give counterparts better insight into which steps in the design workflow need to be completed before the MVC can move to the development phase.
• We take that prototype through a user testing to validate it is solving the original problem.
• When the solution is validated with users - it is time to break the big solution down into smaller issues involving a whole team (PM, Engineers, Product Designer, QA and Technical Writer).
• Engineers, Product Designer, EM and PM are expected to work closely together to find the most technically feasible and smallest feature set to deliver value to early customers and provide feedback for future product development. Check out iteration strategies for help.

We aim to design broadly for an epic or full feature at least one milestone ahead of time and then break the big solution into smaller issues to pick up in the next milestones. Suppose working one milestone ahead to design the big solution is not possible. In that case, Engineering and Product Designer will define the first most technically feasible and smallest feature set (MVC) to satisfy early customers that will be implemented in the same milestone.

Reviewing Merge Requests

UX should not be seen as a "gate" for reviewing Merge Requests. To avoid blocking reviews and speed up the response time, Product Designer and the Engineering DRI should work collaboratively throughout the feature development process and check-in often, so the UX approval at the stage of reviewing merge request is not required.

Tips to avoid blocking reviews:

• Product Designers should point Engineers to the needed patterns to ensure product consistency and speed up development time. (example: Pajamas components for Frontend engineers, API documentation for Back-end engineers)
• Whenever Engineering DRI submits a Merge Request that is causing changes to UI or user experience - Engineering DRI should tag the appropriate Product Designer for visibility. However, no UX approval on Merge Requests is expected at the review stage (except when the funtionality was not designed by the UX department).

Dealing with Community Merge Requests

For more details on how to contribute to GitLab generally, please see our documentation.

Aligning on feature development

The Engineering DRI works with the Product Designer throughout the workflow:in dev phase to uncover possible problems with the solution early enough that exhibit unexpected behaviour to what was originally agreed upon. If there are changes to be added that weren't agreed upon in the initial issue - a followup issue should be made and the Engineering DRI should work with the Product Manager to schedule that issue in a following milestone. This allows us to focus on cleanup over signoff, iterate quickly on issues with a low level of shame, and still make sure we accomplish what we've agreed upon. We should be careful not to hold off on completing these followup issues so that we don't build up a significant amount of UX debt issues.

If we find that solutions are consistently not matching the agreed upon design, we will hold a retrospective with the DRI, designer, and product manager to discuss where the gaps in communication are so that we can improve. It may be necessary to begin requiring a UX approval for merge requests on certain issues to help the Engineering DRI meet the requirements.

Avoiding crunch times between UX and Engineering

• Ideally, the Product Manager works 3 months ahead of time with a Product Designer to validate the problem and work on the solution. See Validation track for more details. This allows us to come up with the bigger idea ahead of time, and work further with Engineering to break it down into smaller iterations. Ideally, this should be completed before the implementation milestone starts.
• Product Designer, PM, and Engineering use the Design phase in the Validation track to talk about complexities and discuss challenges and uncover blockers. Once we are all in agreement - we can have a put a direction on the board - see Solution Validation phase for details.
• Engineers and Product Designers should stay in contact and frequently align throughout the Build track to avoid unplanned changes.

Technical debt

We track our technical debt using the following CI Technical Debt issue board, where we track issues in the planning phase.

This board has 2 main sections:

1. Issue readiness
   • In workflow::planning breakdown we find issues that are currently being groomed.
   • In workflow::ready for development we find issues that clearly defined and a weight has been assigned.
2. Impact breakdown. We use severity::1, severity::2, severity::3 and severity::4 labels to classify the impact of the specific tech debt item. We use the list below as a guideline to grade the impact.
   • severity::1 Blocking or Critical
     • blocking debt that prevents any further changes in the area
     • we stumble on the same complex code every single time and it causes serious slow down on development
     • the problem exists in a core part of our domain that represents a dependency for a large number of features
     • the problem is wild spread throughout several domains
     • the problem is related to a number of reported severity::1 bugs
   • severity::2 High
     • blocks many changes in the area or makes them difficult to work around it
     • the problem exists in an area that changes frequently, causing it or workarounds to spread easily
     • the problem is related to an important feature (e.g multi-project pipelines)
     • the problem is very common in a specific domain and leaks into other domains too
     • the problem is related to a number of reported severity::2 bugs
   • severity::3 Medium
     • the problem exists in a feature that has a supporting role in our domain
     • the problem exists in an area that does not change frequently
     • the problem is very common in a specific domain but limited to its domain boundaries
     • workarounds slow down development
     • the problem is related to a number of reported severity::3 bugs
   • severity::4 Low
     • the problem is very isolated and has low or no impact to development
     • the area affected does not change frequently
     • the problem is related to a number of reported severity::4 bugs

Note that a multiple factors can exist at once. In that case use your judgment to either bump the impact score or lower it. For example:

• the problem exists in a feature that has a supporting role in our domain but it's related to a number of severity::2 bugs. Then choose severity::2.
• the problem is related to an important feature but the workaround is acceptable as the code around it does not change frequently. Then choose severity::3.

Retrospectives

The team leverages a monthly async retrospective process as a way to celebrate success and look for ways to improve. The process for these retrospectives aligns with the automated retrospective process used by many teams at GitLab. The process is defined here: https://gitlab.com/gitlab-org/async-retrospectives#how-it-works.

A new retrospective issue is created on the 27th of each month, and remains open until the 26th of the following month. Team members are encouraged to add comments to that issue throughout the month as things arise as a way to capture topics as they happen. The current issue can be found in https://gitlab.com/gl-retrospectives/verify/-/issues.

On the 16th of each month a summary of the milestone will be added to the issue and the team will be notified to add any additional comments to the issue.

As comments are added to the issue, team members are encourage to upvote any comments they feel are important to callout, and to add any additional discussion points as comments to the original thread.

Around the 26th of the month, or after the discussions have wrapped up the backend engineering manager will summarize the retrospective and create issues for any follow up action items that need to be addressed. They will also redact any personal information, customer names, or any other notes the team would like to keep private. Once that has been completed the issue will be made non-confidential and closed.

Team Communication

The team is globally distributed and separated by many timezones. This presents some challenges in how we communicate since our work days only overlap by a couple hours. We have decided as a team to embrace asynchronous communication because scheduling meetings is difficult to coordinate. We meet as a team one day per week, on Wednesdays for a quick team meeting and 1-on-1s.

• We are pro-active about asking questions in advance with the understanding that the turnaround on receiving an answer is usually a full working day.
• Our engineering plan and issue boards need to be updated regularly to communicate what stage our work is in.
• Any meetings that we hold need a preset agenda and are recorded for folks who are unable to attend.
• Having a positive work/life balance, despite these challenges, must be a high priority and handled intentionally so no one is routinely starting early or staying late to attend meetings.

How to work with us

Slack

Daily standup updates are posted to #g_ci.

Feel free to ask us questions directly in this Slack channel and someone will likely get back to you within 24 hours (by the next business day). We will use following emojis to respond to the posted question accordingly:

• – :eyes: to indicate that one of us has seen it
• – :white_check_mark: to indicate that the question has been answered

The verify stage has a separate Slack channel under #s_verify, which encompasses the other teams of Verify.

GitLab Issues and MRs

Most spontaneous team communiation happens in issues and MRs. Our issues have a group label of ~"group::continuous integration". You can also tag a team member with @mention in the issue if you have someone specific to ask.

If you need to call the attention of the entire group, you can tag @gitlab-com/ci-group.

Developer Onboarding

Welcome to the team! Whether you're transferring internally or joining GitLab as a new hire, you'll be assigned an onboarding buddy. If you are an internal transfer, your manager will appoint you an onboarding buddy so you can have someone to work with as you're getting familiarized with our codebase, our tech stack and the team.

Read over this page as a starting point, and feel free to set up regular sync or async conversations with your buddy. It's also recommended to schedule a few coffee chats to meet some members of our team. Your manager may optionally create an onboarding checklist for you to go through if it's helpful.

In the our list of issues, there are also ones with an ~onboarding label that are smaller issues to onboard onto the product. If there aren't any in the current Workflow issue board to work on, reach out to your EM/PM to see which issues you can work on as part of your onboarding period.