Fulfillment Sub-department

1. You are here:
2. Engineering
3. Development Department
4. Fulfillment Sub-department

Maintained by:

On this page

• Vision
• Mission
• Direction
  • OKRs
• Team members
  • Fulfillment Sub-department People Leaders
  • Fulfillment:Purchase BE Team
  • Fulfillment:Purchase FE Team
  • Fulfillment:License BE Team
  • Fulfillment:License FE Team
  • Fulfillment:Utilization BE Team
  • Fulfillment:Utilization FE Team
• Stable counterparts
  • Fulfillment Product Management
  • Fulfillment Quality Engineering
  • Fulfillment UX
  • Fulfillment Security
  • Enterprise Applications
  • Sales Ops
  • Enterprise Sales
  • Commercial Sales
  • Channel Sales
  • Billing and Accounts Receivable
• How we work
  • Planning
    • Purchase
    • License
    • Utilization
  • Estimation
    • Estimation Template
  • Picking something to work on
    • Organizing the work
  • Weekly async issue updates
  • Quality
    • Quad-planning workflow
    • Tests
  • Deployment
  • Release of Revenue Impacting Changes
    • CustomersDot
    • LicenseDot
    • Escalation Process
    • Feature freeze
  • Measuring Engineering Performance
  • CustomersDot Engineer:Maintainer ratio
  • Meetings (Sync)
  • Retrospectives
  • Onboarding
• Iteration Retrospectives
  • Frequency
  • Preparation
  • Participation
  • Template (optional)
• Performance indicators
• Knowledge sharing
• Common links

Vision

A high performing team who provide our customers with a world-class buyer experience through the products we build. Our team strives to build an experience that is delightful, performant, trustworthy, and reliable.

Mission

Fulfillment focuses on improving our capabilities and metrics in the following areas:

• Fulfillment: Direction, Group Responsibilities, Fulfillment Development Team
• Purchase: Direction, Group Responsibilities, Backend Team, Frontend Team
• License: Direction, Group Responsibilities, Backend Team, Frontend Team
• Utilization: Direction, Group Responsibilities, Backend Team, Frontend Team

Direction

In addition to the Fulfillment Product Direction, the Fulfillment Development Sub-department strives to:

• Increase the reliability and availability of our Fulfillment infrastructure
• Make foundational technical improvements to the architecture and data models of the Fulfillment systems
• Improve the developer experience of Fulfillment engineers through better tooling and documentation

OKRs

We follow the OKR (Objective and Key Results) framework to set and track goals on a quarterly basis. The Fulfillment section OKRs are set across the enture Quad:

• FY22-Q3 Quad OKR's
• FY22-Q4 Quad OKR's

Team members

Fulfillment Sub-department People Leaders

Fulfillment:Purchase BE Team

Fulfillment:Purchase FE Team

Fulfillment:License BE Team

Fulfillment:License FE Team

Fulfillment:Utilization BE Team

Fulfillment:Utilization FE Team

Stable counterparts

Fulfillment Product Management

Fulfillment Quality Engineering

Fulfillment UX

Technical Writing

Fulfillment Security

Enterprise Applications

Sales Ops

Enterprise Sales

Commercial Sales

Channel Sales

Billing and Accounts Receivable

How we work

• In accordance with our GitLab values
• Transparently: nearly everything is public, we record/livestream meetings whenever possible
• We get a chance to work on the things we want to work on
• Everyone can contribute; no silos

Planning

We plan in monthly cycles in accordance with our Product Development Timeline. Release scope for an upcoming release should be finalized by the 1st.

On or around the 26th: Product meets with Engineering Managers for a preliminary issue review. Issues are tagged with a milestone and are estimated initially.

Purchase

• 13.11 Board
• 13.11 Frontend Board
• 13.11 Planning Issue

License

• 13.11 Board
• 13.11 Planning Issue

Utilization

• 13.11 Board
• 13.11 Planning Issue

Estimation

Before work can begin on an issue, we should estimate it first after a preliminary investigation. This is normally done in the monthly planning meeting.

In planning and estimation, we value velocity over predictability. The main goal of our planning and estimation is to focus on the MVC, uncover blind spots, and help us achieve a baseline level of predictability without over optimizing. We aim for 70% predictability instead of 90%. We believe that optimizing for velocity (MR throughput) enables our Fulfillment teams to achieve a weekly experimentation cadence.

• If an issue has many unknowns where it's unclear if it's a 1 or a 5, we will be cautious and estimate high (5).
• If an issue has many unknowns, we can break it into two issues. The first issue is for research, also referred to as a Spike, where we de-risk the unknowns and explore potential solutions. The second issue is for the implementation.
• If an initial estimate is incorrect and needs to be adjusted, we revise the estimate immediately and inform the Product Manager. The Product Manager and team will decide if a milestone commitment needs to be adjusted.

Estimation Template

The following is a guiding mental framework for engineers to consider when contributing to estimates on issues.

### Refinement / Weighting

**Ready for Development**: Yes/No

<!--
Yes/No

Is this issue sufficiently small enough, or could it be broken into smaller issues? If so, recommend how the issue could be broken up.

Is the issue clear and easy to understand?
-->

**Weight**: X

**Reasoning**:

<!--
Add some initial thoughts on how you might breakdown this issue. A bulleted list is fine.

This will likely require the code changes similar to the following:

- replace the hexdriver with a sonic screwdriver
- rewrite backups to magnetic tape
- send up semaphore flags to warn others

Links to previous example. Discussions on prior art. Notice examples of the simplicity/complexity in the proposed designs.
-->


**MR Count**: Y
<!--
- 1 MR to update the driver worker
- 1 MR to update docs regarding mag tape backups

Let me draw your attention to potential caveats.
-->

**Testing considerations**:
<!--
- ensure that rotation speed of sonic screwdriver doesn't exceed rotational limits
-->

Picking something to work on

Engineers can find and open the milestone board for Fulfillment and begin working first on those with the deliverable label.

It's possible for engineers to pick any of the remaining issues for the milestone once the deliverables are done. If the engineer has no preference, they can choose the next available issue from the top.

The following table will be used as a guideline for scheduling work within the milestone:

Organizing the work

We generally follow the Product Development Flow:

1. workflow::problem validation - needs clarity on the problem to solve
2. workflow::design - needs a clear proposal (and mockups for any visual aspects)
3. workflow::solution validation - designs need to be evaluated by customers, and/or other GitLab team members for usability and feasibility
4. workflow::planning breakdown - needs a Weight estimate
5. workflow::scheduling - needs a milestone assignment
6. workflow::ready for development
7. workflow::in dev
8. workflow::in review
9. workflow::verification - code is in production and pending verification by the DRI engineer

Generally speaking, issues are in one of two states:

• Discovery/refinement: we're still answering questions that prevent us from starting development,
• Implementation: an issue is waiting for an engineer to work on it, or is actively being built.

Basecamp thinks about these stages in relation to the climb and descent of a hill.

While individual groups are free to use as many stages in the Product Development Flow workflow as they find useful, we should be somewhat prescriptive on how issues transition from discovery/refinement to implementation.

Weekly async issue updates

Every Friday, each engineer is expected to provide a quick async issue update by commenting on their assigned issues using the following template:

<!---
Please be sure to update the workflow labels of your issue to one of the following (that best describes the status)"
- ~"workflow::In dev"
- ~"workflow::In review"
- ~"workflow::verification"
- ~"workflow::blocked"
-->
### Async issue update
1. Please provide a quick summary of the current status (one sentence).
1. When do you predict this feature to be ready for maintainer review?
1. Are there any opportunities to further break the issue or merge request into smaller pieces (if applicable)?
1. Were expectations met from a previous update? If not, please explain why.

We do this to encourage our team to be more async in collaboration and to allow the community and other team members to know the progress of issues that we are actively working on.

Quality

GitLab's Quality is everyone's responsibility. The SETs embedded within Fulfillment section follow the Quality Engineering Departments's principles primarily to ensure that everyone is aware of product quality.

Quad-planning workflow

The SETs helps facilitate the quad-planning process. This is the participation of Product Management, Development, UX, and the Quality team which aims to bring test planning as a topic before the development of any feature.

We follow the Quad Planning process defined here.

Quad Planning Dashboard showcases the total Planned issues for Quad Planning vs the actual ones for each milestone.

Tests

The CustomersDot has different types of tests running:

1. Linting and rubocop jobs
2. Unit tests (specs, these could be of many types, such as controller specs)
3. Integration tests (specs, mocking external calls)
4. Frontend tests
5. E2E integration tests via Watir

We also have a flag VCR that mocks external calls to Zuora by default. We have a daily pipeline that runs at 9AM UTC with the flag set so the API calls hit the Zuora sandbox and we are notified of any failure (due to potential API changes).

Any test failure is notified to #s_fulfillment_status including a link to the pipeline. Pipeline failures will prevent deployments to staging and production.

Deployment

Release of Revenue Impacting Changes

Fulfillment is unique in Engineering because our changes can directly impact revenue. Changes which impact revenue or are otherwise high risk may require broader scrutiny including input from teams outside of Fulfillment. PM is the DRI on these decisions, taking input from stakeholders including Sales, Enterprise Applications, Marketing, Finance, and Customer Support.

We aspire to be as iterative as possible including even when a change potentially poses high risk.

Below are examples of changes that can be high risk because they can require coordination across many teams:

• Pricing changes
• Billing changes
• Launching a new paid feature
• Deprecating an existing paid feature
• Changes to terms of service and related contract changes
• Changes to the way the consumption of resources is calculated or displayed (e.g., user counts, CI minutes)

Below are changes that can often be low risk:

• Backend enhancements that are not yet used by the frontend
• Frontend and backend Enhancements that are behind a feature flag
• Frontend enhancements that are not behind a feature flag, but are labeled as "beta" or "experimental"

The following processes should be followed for high-risk changes:

• Update feature documentation of the new/updated behavior and share with relevant stakeholders
• Use confidential issues for these changes
• Release changes under a feature flag so this can be enabled on the release date
• Create a rollout issue for the release date and make sure all parties are informed about the process. Example
• Consider enabling this to a subset of customers so changes can be tested iteratively
• Use the regular MR process even if the issue is confidential until confidential MRs are improved (1, 2)

CustomersDot

We use CD (Continuous Deployment) for CustomersDot and a MR goes through the following stages once it gets merged into the staging branch:

If something goes wrong at the Verification stage, we could create an issue with the label production::blocker, which will prevent deployment to production. The issue cannot be confidential.

For MRs with significant changes, we should consider using feature flags or create an issue with the production::blocker label to pause deployment and a allow for longer testing.

LicenseDot

We use an automatic deployment to staging, but manual deployment to production for LicenseDot.

Maintainers of the application need to trigger a manual action on the master branch in order to deploy to production.

The app lives at https://license.gitlab.com

The staging environment can be found here

Escalation Process

In most cases an MR should follow the standard process of review, maintainer review, merge, and deployment as outlined above. When production is broken:

1. First determine whether the Rapid Engineering Response process can be followed. This will depend on availability and how critical the situation is.
2. The three hour wait between Staging and Production auto deploy can be bypassed with a manual deployment by a Maintainer.
3. When there are no project maintainers available (CustomersDot, LicenseDot) an additional GitLab Team Member with Maintainer access can be asked to assist.

In these cases please ensure:

1. There is an issue describing the reasons for the escalation, as per the Rapid Engineering Response. Consider '@'mentioning the relevant Growth/Fulfillment teams.
2. The change is announced on the #s_fulfillment channel.

Feature freeze

The feature freeze for Fulfillment occurs at the same time as the rest of the company, normally around the 18th.

(*) feature freeze may vary according to the auto-deploy document.

Any issues not merged on the current milestone post feature freeze, will need to be moved to the next one (priority may also change for those).

Measuring Engineering Performance

See the Fulfillment Section Performance Indicators as well as the Centralized Engineering Dashboards.

CustomersDot Engineer:Maintainer ratio

Meetings (Sync)

We hold optional synchronous social meetings weekly where we chat about anything outside work. These meetings happen each Wednesday alternating between 10:00am UTC (EMEA) and 4:00pm (AMER). Check the Fulfillment Google calendar for more information.

Retrospectives

After the 8th, the Fulfillment team conducts an asynchronous retrospective. You can find current and past retrospectives for Fulfillment in https://gitlab.com/gl-retrospectives/fulfillment/issues/.

Onboarding

To be added

Iteration Retrospectives

There are several resources at GitLab for improving our iteration capability:

• engineering iteration documentation
• product iteration documentation
• self-guided iteration training issue
• iteration office hours
• milestone retrospectives
• iteration retrospectives

Agile software development practices uses the phrase Iteration Retrospective to describe what we perform during our Milestone Retrospective. While the Milestone Retrospective is essential in understanding how we can improve, it does have a broad focus across multiple teams. It's a safe space where everyone can contribute through reflection on what we can continue, what we can correct, and what we can improve.

An Iteration Retrospective, in the GitLab values sense that we're using here, is more focused on a specific example (or few examples) of a successful or unsuccessful iteration attempt. That is, an Issue, Epic, or merge request that we can examine deeply to understand what we can do differently in the future to be more iterative and efficient. These retrospectives will provide a growth opportunity to work on developing an iterative mindset - build it, then build it better.

Each group should perform their own Iteration Retrospective, but share as transparently as possible to help fellow teammates and other teams learn from your investigation.

Frequency

Iteration Retrospectives should be conducted quarterly.

Preparation

Engineering Manager(s) or Other team members should review current or previous milestones to find a good, challenging candidate issue for the Iteration Retrospective. Choose one (or two if related) from the suggestions, then create a retrospective issue.

Review these rules for an effective retrospective.

Participation

Everyone can and should contribute asynchronously to the discussion. Present your ideas on alternative methods or mechanisms to break down the problem stated in chosen issue for review. If the problem breakdown was fine, suggestions about process or other refinements are also welcome.

Don't belabor the point, participation should be timeboxed to 1 week. A 1 week due date quick action is available in the template below.

Consider handbook updates, process changes, and bubbling up information from your conclusions after this retrospective.

Template (optional)

Iteration is one of six GitLab Values, but also really difficult. By focusing how we have iterated well in the past and how we have not will help us iterate faster. Please contribute the following [Iteration Retrospective](#link-to-handbook-page).

## Summary

<!--
Include a brief summary of the issue. Consider including key outcomes, significant changes, and impact.
-->

`Summary of the Issue`

## Details

<!--
Complete the following details when creating the issue.
-->

- Did the team meet the iteration goals? Why or why not?
- How many MRs were created to complete the Iteration?
- What were the Days to Merge for the MRs related to the this issue?
- Are there successes or opportunities for improvement with respect to collaboration with peers or stable counterparts?
- Could the original issue have been broken down into smaller components?
- Could the MR(s) have been broken down into smaller components?

## Tasks

**Read**

- [Iteration Value](https://about.gitlab.com/handbook/values/#iteration)
- [Engineering Iteration](https://about.gitlab.com/handbook/engineering/workflow/iteration/)
- [Why iteration helps increase the merge request rate](https://about.gitlab.com/blog/2020/05/06/observations-on-how-to-iterate-faster/)

**Watch**

- [Interview about iteration in engineering with Christopher and Sid](https://www.youtube.com/watch?v=tPTweQlBS54)
- [Iteration Office Hours with CEO](https://www.youtube.com/watch?v=liI2RKqh-KA)


**Contribute**

In the threads below consider the following:

- Why was it (un)successful? If successful, how did it meet the definition of an [MVC](https://about.gitlab.com/handbook/product/product-principles/#the-minimal-viable-change-mvc)?
- How did the example not meet the definition of an [MVC](https://about.gitlab.com/handbook/product/product-principles/#the-minimal-viable-change-mvc)? In what ways could you have iterated differently? List specific examples.
- Identify areas of improvement for the team that can be incorporated into our processes and workflows.

Follow [these rules](https://about.gitlab.com/handbook/engineering/management/group-retrospectives/) for an effective retrospective.

Check your name off the completed list when all tasks are complete.

**Completed**

<!--
When creating this issue replace the person placeholders with teamembers and select stable counterparts as you see fit.
-->

- [ ] Person
- [ ] Person
- [ ] Person
- [ ] Person

**Next Steps**

- [ ] Engineering Manager to update your team's handbook page with ideas and improvements you've incorporated to improve iteration.

## Thread Prompts

<!--
Use these prompts or similar to start conversation threads.
-->

1. What aspects of the Issue/Epic and resulting MR(s) were very good/bad examples of Iteration and why?
1. Does anyone have alternative ideas for how this work could have been broken down?
1. Is there anything that we can change in our processes or the way we work that could improve our iteration as a team?


/due in 1 week
/label ~"Iteration Retrospective"

Performance indicators

To be added

Knowledge sharing

• Generating test licenses, adding them to GitLab, and then finding and removing them from the GitLab database
  • Document with notes
  • Recording
• New Integration with Sentry
  • Document with notes
  • Recording
• Intro to FE GraphQL at GitLab with Vue Apollo
  • Slides
  • Recording
• CustomersDot Resource Video Library
• Zuora University Select Topics
• Confidential Issue Git Pre-push Hook to prevent any accidental push to any non-Gitlab Security remote whenever a developer works on a confidential issue. Reference: How to customize Git Hooks.

Common links

• Product Vision
• Fulfillment section
• InfraDev Team initiative
• Fulfillment Software Architecture
• All open Fulfillment epics
• Issue Tracker
• Slack channel #s_fulfillment
• Daily standup channel #s_fulfillment_daily
• Team calendar
• Fulfillment technical debt status