Fulfillment Sub-department

The Fulfillment Sub-department is composed of the Billing and Subscription Management, Fulfillment Platform, Provision, Purchase, and Utilization development teams working on the infrastructure between the systems which affect the user purchasing process that support the GitLab DevOps Platform.

Vision

A high performing team who provides our customers with a world-class buyer experience through the products we build. Our team strives to build an experience that is delightful, performant, trustworthy, and reliable.

Mission

Fulfillment focuses on improving our capabilities and metrics in the following areas:

Direction

In addition to the Fulfillment Product Direction, the Fulfillment Development Sub-department strives to:

  • Increase the reliability and availability of our Fulfillment infrastructure
  • Make foundational technical improvements to the architecture and data models of the Fulfillment systems
  • Improve the developer experience of Fulfillment engineers through better tooling and documentation

Team members

Fulfillment Sub-department People Leaders

Name Role
Chase SouthardChase Southard Engineering Manager, Fulfillment:Utilization
Isabel SandinIsabel Sandin Engineering Manager, Fulfillment:Provision
James LopezJames Lopez Senior Engineering Manager, Fulfillment:Fulfillment Platform
Jerome NgJerome Ng Director of Engineering, Fulfillment, Interim Strategy and Operations Lead, JiHu
Ragnar HardarsonRagnar Hardarson Engineering Manager, Fulfillment:Subscription Management Team
Shreyas AgarwalShreyas Agarwal Software Engineering Manager, Fulfillment:Purchase

Fulfillment:Billing and Subscription Management Team

Name Role

Fulfillment:Purchase Team

Name Role
Shreyas AgarwalShreyas Agarwal Software Engineering Manager, Fulfillment:Purchase

Fulfillment:Provision Team

Name Role
Bishwa Hang RaiBishwa Hang Rai Senior Backend Engineer, Fulfillment:Purchase
Corinna GogolokCorinna Gogolok Staff Backend Engineer, Fulfillment:Provision
Courtney MeddaughCourtney Meddaugh Product Manager, Fulfillment:Provision
Isabel SandinIsabel Sandin Engineering Manager, Fulfillment:Provision
Mohamed HamdaMohamed Hamda Backend Engineer, Fulfillment:Provision
Mohamed MoustafaMohamed Moustafa Fullstack Engineer
Paulo BarrosPaulo Barros Senior Backend Engineer, Fulfillment:Provision
Qingyu ZhaoQingyu Zhao Senior Backend Engineer, Fulfillment:Provision
Vamsi VempatiVamsi Vempati Senior Frontend Engineer, Fulfillment:Provision

Fulfillment:Utilization Team

Name Role
Alexandra MartinAlexandra Martin Sr. PM, Fulfillment:Purchase
Ammar AlakkadAmmar Alakkad Senior Frontend Engineer, Fulfillment:Utilization
Angelo GulinaAngelo Gulina Senior Frontend Engineer, Fulfillment:Utilization
Chase SouthardChase Southard Engineering Manager, Fulfillment:Utilization
Jason GoodmanJason Goodman Senior Backend Engineer, Fulfillment:Utilization
Kos PalchykKos Palchyk Senior Frontend Engineer, Fulfillment:Utilization
Sheldon LedSheldon Led Senior Frontend Engineer, Fulfillment::Utilization
Suraj TripathiSuraj Tripathi Senior Backend Engineer, Fulfillment:Utilization
Vijay HawoldarVijay Hawoldar Staff Backend Engineer, Fulfillment:Utilization

Fulfillment:Fulfillment Platform Team

Name Role
Aishwarya SubramanianAishwarya Subramanian Senior Backend Engineer, Fulfillment:Fulfillment Platform
Etienne BaquéEtienne Baqué Senior Backend Engineer, Fulfillment:Fulfillment Platform
James LopezJames Lopez Senior Engineering Manager, Fulfillment:Fulfillment Platform
Tarun VellishettyTarun Vellishetty Fullstack Engineer, Fulfillment:Fulfillment Platform
Tyler AmosTyler Amos Staff Backend Engineer, Fulfillment:Fulfillment Platform
Vitaly SlobodinVitaly Slobodin Staff Frontend Engineer, Fulfillment:Fulfillment Platform
Vladlena ShumiloVladlena Shumilo Senior Backend Engineer, Fulfillment:Fulfillment Platform

Stable counterparts

The following members of other functional teams are our stable counterparts.

Name Role
Aishwarya SubramanianAishwarya Subramanian Senior Backend Engineer, Fulfillment:Fulfillment Platform
Alexandra MartinAlexandra Martin Sr. PM, Fulfillment:Purchase
Aman LuthraAman Luthra Backend Engineer, Fulfillment:Subscription Management Team
Ammar AlakkadAmmar Alakkad Senior Frontend Engineer, Fulfillment:Utilization
Angelo GulinaAngelo Gulina Senior Frontend Engineer, Fulfillment:Utilization
Bishwa Hang RaiBishwa Hang Rai Senior Backend Engineer, Fulfillment:Purchase
Chase SouthardChase Southard Engineering Manager, Fulfillment:Utilization
Corinna GogolokCorinna Gogolok Staff Backend Engineer, Fulfillment:Provision
Costel MaximCostel Maxim Senior Security Engineer, Application Security, Plan (Project Management, Product Planning, Certify), Create:Source Code, Growth, Fulfillment:Purchase, Fulfillment:Provision, Fulfillment:Utilization, Systems:Gitaly
Courtney MeddaughCourtney Meddaugh Product Manager, Fulfillment:Provision
Diana ZubovaDiana Zubova Frontend Engineer, Fulfillment:Subscription Management
Etienne BaquéEtienne Baqué Senior Backend Engineer, Fulfillment:Fulfillment Platform
Isabel SandinIsabel Sandin Engineering Manager, Fulfillment:Provision
James LopezJames Lopez Senior Engineering Manager, Fulfillment:Fulfillment Platform
Jason GoodmanJason Goodman Senior Backend Engineer, Fulfillment:Utilization
Jerome NgJerome Ng Director of Engineering, Fulfillment, Interim Strategy and Operations Lead, JiHu
Josianne HysonJosianne Hyson Backend Engineer, Fulfillment:Subscription Management
Kos PalchykKos Palchyk Senior Frontend Engineer, Fulfillment:Utilization
Laura CallahanLaura Callahan Frontend Engineer, Fulfillment:Subscription Management
Mohamed HamdaMohamed Hamda Backend Engineer, Fulfillment:Provision
Michael LunøeMichael Lunøe Staff Frontend Engineer, Fulfillment:Subscription Management
Minahil NicholsMinahil Nichols Fullstack Engineer, Fulfillment:Subscription Management
Omar FernandezOmar Fernandez Director of Product Management, Fulfillment
Ottilia WesterlundOttilia Westerlund Security Engineer, Fulfillment (Fulfillment Platform, Subscription Management), Govern (Security Policies, Threat Insights), Monitor (Observability), Plan (Product Planning), AI-powered:AI Framework
Paulo BarrosPaulo Barros Senior Backend Engineer, Fulfillment:Provision
Qingyu ZhaoQingyu Zhao Senior Backend Engineer, Fulfillment:Provision
Ragnar HardarsonRagnar Hardarson Engineering Manager, Fulfillment:Subscription Management Team
Richard ChongRichard Chong Senior Software Engineer in Test, Test Engineering, Fulfillment section
Ryan CobbRyan Cobb Backend Engineer, Fulfillment:Subscription Management
Sheldon LedSheldon Led Senior Frontend Engineer, Fulfillment::Utilization
Shreyas AgarwalShreyas Agarwal Software Engineering Manager, Fulfillment:Purchase
Sharmad NachnolkarSharmad Nachnolkar Senior Frontend Engineer, Fulfillment:Subscription Management
Suraj TripathiSuraj Tripathi Senior Backend Engineer, Fulfillment:Utilization
Tatyana GolubevaTatyana Golubeva Principal Product Manager, Fulfillment:Subscription Management
Tarun VellishettyTarun Vellishetty Fullstack Engineer, Fulfillment:Fulfillment Platform
Tyler AmosTyler Amos Staff Backend Engineer, Fulfillment:Fulfillment Platform
Valerie BurtonValerie Burton Senior Software Engineer in Test, Test Engineering, Fulfillment section
Vijay HawoldarVijay Hawoldar Staff Backend Engineer, Fulfillment:Utilization
Vitaly SlobodinVitaly Slobodin Staff Frontend Engineer, Fulfillment:Fulfillment Platform
Vladlena ShumiloVladlena Shumilo Senior Backend Engineer, Fulfillment:Fulfillment Platform
Vamsi VempatiVamsi Vempati Senior Frontend Engineer, Fulfillment:Provision

Fulfillment Security

Name Role
Andrew KellyAndrew Kelly Security Manager, Application Security

Enterprise Applications

Sales & Go-To-Market (GTM)

The Sales & Go-To-Market stable counterpart will serve as the GTM DRI for strategic cross-functional intiatives, new feature and system updates, and bug resolution. They will ensure that an end-to-end approach is taken into account when communicating with the field and users. Fulfillment PMs can engage the counterpart for new GTM work by using @mention in GitLab as the primary method to submit and prioritize tasks.

Billing and Accounts Receivable

Name Role
Sarah McCauleySarah McCauley Director, Billing & Accounts Receivable
Anna PiasecznaAnna Piaseczna Senior Manager, Billing Operations GPO

Customer Success

Support Engineering

Name Role
John LyttleJohn Lyttle Manager, Support Engineering
Michael DunningerMichael Dunninger Support Engineering Manager (Americas North)
Ket SlaatsKet Slaats Manager, Support Engineering (APAC)

Chief of Staff Team to the CEO

Name Role
Ian PedowitzIan Pedowitz Director, Strategy and Operations

Project management process

  • In accordance with our GitLab values
  • Transparently: nearly everything is public, we record/livestream meetings whenever possible
  • We get a chance to work on the things we want to work on
  • Everyone can contribute; no silos

SAFE

Working in SAFE manner at GitLab is everyone’s responsibility. We along with our stable counterparts in Sales and Billing contribute to an area of the product that potentially could encounter sensitive or financial information, and could have an effect on the business as a whole. Therefore, it’s important for Fulfillment team members to ensure that the SAFE epics, issues, videos, MRs, and other artifacts we produce are kept confidential.

On occasion, it may be prudent to include language like the following to the description of public issues where potentially SAFE discussions are happening.

This page may contain information related to upcoming products, features and functionality. It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes. Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc.

Similarly, not all information should be included in the public handbook. Instead, use the private internal handbook for this SAFE information.

Please reference this documentation about promising features in future versions for more information.

Planning

We plan in monthly cycles in accordance with our Product Development Timeline. Release scope for an upcoming release should be finalized by the 1st.

On or around the 26th: Product meets with Engineering Managers for a preliminary issue review. Issues are tagged with a milestone and are estimated initially.

Planning Issues

Intake Request

To request work to be added to the Fulfillment Roadmap, please open an issue by following this link and tag one of the Fulfillment Product Managers.

A Fulfillment Product Manager will be assigned to review the request and start evaluating the scope and impact. This may take some time depending on team capacity. Once evaluated, the PM will:

  1. Create a new Epic, capturing relevant details from the intake issue.
  2. Add this new Epic to the Fulfillment roadmap.
  3. Update the intake issue description with a link to the new epic.
  4. Close the intake issue.

We strictly adhere to this Intake Request process to ensure we capture the full details of a request before actioning on it. Once our Product Managers receive this request, we’ll work with the requestor to ensure we have a full understanding of the problem and requirements. After this, we will prioritize your request.

Prioritization

We follow the prioritization guidelines and use cross-functional prioritization. In addition to these company-wide prioritization inputs, PMs review the L&R Support Priority Issues list for Fulfillment.

Every team uses the monthly prioritization template for cross-functional dashboard reviews every month.

Fulfillment stage dashboards

Purchase team dashboards

Subscription Management team dashboards

Provision team dashboards

Utilization team dashboards

Fulfillment Platform team dashboards

Estimation

Before work can begin on an issue, we should estimate it first after a preliminary investigation. This is normally done in the monthly planning meeting.

Weight Description (Engineering)
1 The simplest possible change. We are confident there will be no side effects.
2 A simple change (minimal code changes), where we understand all of the requirements.
3 A simple change, but the code footprint is bigger (e.g. lots of different files, or tests effected). The requirements are clear.
5 A more complex change that will impact multiple areas of the codebase, there may also be some refactoring involved. Requirements are understood but you feel there are likely to be some gaps along the way.
8 A complex change, that will involve much of the codebase or will require lots of input from others to determine the requirements.
13 A significant change that may have dependencies (other teams or third-parties) and we likely still don’t understand all of the requirements. It’s unlikely we would commit to this in a milestone, and the preference would be to further clarify requirements and/or break in to smaller Issues.

In planning and estimation, we value velocity over predictability. The main goal of our planning and estimation is to focus on the MVC, uncover blind spots, and help us achieve a baseline level of predictability without over optimizing. In general departments at GitLab aim for [70% predictability] but in the Fulfillment sub-department we aim for 80% predictability since our work is typically cross-functional and we need to be in lockstep with other departments.

  • If an issue has many unknowns where it’s unclear if it’s a 1 or a 5, we will be cautious and estimate high (5).
  • If an issue has many unknowns, we can break it into two issues. The first issue is for research, also referred to as a Spike, where we de-risk the unknowns and explore potential solutions. The second issue is for the implementation.
  • If an initial estimate is incorrect and needs to be adjusted, we revise the estimate immediately and inform the Product Manager. The Product Manager and team will decide if a milestone commitment needs to be adjusted.

Estimation Template

The following is a guiding mental framework for engineers to consider when contributing to estimates on issues.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
### Refinement / Weighting

<!--
Ready for development means replying yes to the following questions:

- Is this issue sufficiently small enough? If not, break it into smaller issues
- Is it assigned to the correct domain (e.g. frontend, backend)? If not, break it into two issues for the respective domains
– Is the issue clear and easy to understand? If not, try asking further clarification questions and update the description once they are received

If more than 2 MRs are needed, consider adding a table like the following to the description (e.g. under `Implementation plan`).

| Description | MR |
|-|-|
|||

It will help track the status.
-->

- [ ] Ready for development
- [ ] Weight is assigned
- [ ] Number of MRs listed
- [ ] Needs testing considerations
- [ ] Needs documentation updates

**Reasoning:**

<!--
Add some initial thoughts on how you might break down this issue. A bulleted list is fine.

This will likely require the code changes similar to the following:

- replace the hex driver with a sonic screwdriver
- rewrite backups to magnetic tape
- send up semaphore flags to warn others

Links to previous examples. Discussions on prior art. Notice examples of the simplicity/complexity in the proposed designs.
-->

Picking something to work on

Engineers can find and open their team’s planning issue to check their milestone board and begin working first on those with the deliverable label.

It’s possible for engineers to pick any of the remaining issues for the milestone once the deliverables are done. If the engineer has no preference, they can choose the next available issue from the top.

Workflow

We generally follow the Product Development Flow and use the workflow labels as defined there.

Generally speaking, issues are in one of two states:

  • Discovery/refinement: we’re still answering questions that prevent us from starting development,
  • Implementation: an issue is waiting for an engineer to work on it, or is actively being built.

Basecamp thinks about these stages in relation to the climb and descent of a hill.

While individual groups are free to use as many stages in the Product Development Flow workflow as they find useful, we should be somewhat prescriptive on how issues transition from discovery/refinement to implementation.

Quad-planning workflow

The SETs helps facilitate the quad-planning process. This is the participation of Product Management, Development, UX, and the Quality team which aims to bring test planning as a topic before the development of any feature.

We follow the Quad Planning process defined here.

User Experience

We strive to provide excellent usability in all of our workflows, creating a balance between user and business needs. Product Designers work closely with Product Managers and Engineers. Visit the Fulfillment User Experience page for details.

In summary,

  • We measure our progres using UX Scorecards and Category Maturity Scorecards. See our UX Health.
  • We follow the Product Designer Workflows.
  • We track user research efforts using UX Problem Validation and UX Solution Validation labels.
  • We weight issues using the UX issue weighting process.
  • We create separate [UX] and [Engineering] issues to aid in tracking issue weights.
  • [UX] issues are the SSOT for designs and should reflect the latest design decisions. Related implementation issues should link to the [UX] issue for the design details to maintain the SSOT.
  • Fulfillment Product Designers participate in GitLab’s reviewer roulette, but at this time only Fulfillment Product Designers should review CustomersDot MRs and GitLab MRs that require connection to CustomersDot (e.g., the GitLab.com purchase flow).

Approving and merging the work

We have approval rules enabled in CustomersDot, so every MR that targets main needs at least one approval (different from the author/committer). The MRs must meet the following criteria:

  • They need to have at least two different reviewers, including one from a maintainer. However, for trivial MRs with no changes in logic one reviewer is sufficient and you can skip the initial review for such changes (for example, minor dependency updates, test fixes, and plain reverts).
  • They must receive at least one approval.
  • A maintainer’s review is required.

In addition to the approval rules, MRs may require additional reviews as suggested by the Danger bot:

  1. Modifications to the DB require database reviewer and maintainer approval
  2. Security-related issues (such as changes to authentication) require a Security review
  3. Changes to the SFDC APIs require review by the Sales team
  4. Changes to the Zuora APIs require review by the EntApps team
  5. Changes to the user experience require a UX Review.

Weekly async issue updates

Every week, each engineer is expected to provide a quick async issue update by commenting on their assigned issues using the following template:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
<!---
Please be sure to update the workflow labels of your issue to one of the following (that best describes the status)"
- ~"workflow::In dev"
- ~"workflow::In review"
- ~"workflow::verification"
- ~"workflow::blocked"
-->
### Async issue update

1. Please provide a quick summary of the current status (one sentence)
    -
1. How confident are you that this will make it to the current milestone?
    - [ ] Not confident
    - [ ] Slightly confident
    - [ ] Very confident
1. Are there any opportunities to further break the issue or merge request into smaller pieces (if applicable)?
    - [ ] Yes
    - [ ] No
1. Were expectations met from a previous update? If not, please explain why.
    - [ ] Yes
    - [ ] No, ___
1. Are the related issues up to date? Please link any missing issues in the epic that could be linked to this issue
    - [ ] Yes
    - [ ] No

/health_status [on_track, needs_attention, at_risk]

We do this to encourage our team to be more async in collaboration and to allow the community and other team members to know the progress of issues that we are actively working on.

Async projects update

This template is to be used for larger project progress status updates to be shared with leadership & cross-functional partners.

Leadership is interested in the following

  1. Is the project progressing? –> Look at % completion week to week
  2. Are there blockers that need to be cleared? –> Look at Risks & Blockers
  3. When can we expect the next event? –> Look at Key dates

This template is a guideline and feel free to alter it to specific project needs. Before deviating from the template keep in mind that leadership looks at status updates across multiple projects. The more conformity there is between projects the easier it is for leadership to accurately understand status.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
## Status update as of XXXX-XX-XX

### Summary

1. **Key Resources**
  * **_TBD_ this section is optional**

2. **% Complete**: `X%`

The following calculations can help inform this %.

// Numerator: Calculate total weight of closed issues (use `defaultWeight` for issues without weight)
const defaultWeight = 1;
[...document.querySelectorAll('.issue.closed')].reduce((memo, el) => {
    const weightText = el.querySelector('.issuable-weight')?.innerText;
    return memo + Number(weightText) || defaultWeight;
}, 0);

// Denominator: Calculate total weight of open and closed issues (use `defaultWeight` for issues without weight)
const defaultWeight = 1;
[...document.querySelectorAll('.issue')].reduce((memo, el) => {
    const weightText = el.querySelector('.issuable-weight')?.innerText;
    return memo + Number(weightText) || defaultWeight;
}, 0);

3. **Status**: `On Track or Behind` (this is determined based on your how your % complete is trending to your key dates -- are you far enough along to hit your key dates?)

4. **Key Dates**:
    | Event | Estimated Timing | Additional Details | % Complete |
    |---|---|---|---|
    | Design complete |  |  |  |
    | Development complete |  |  |  |
    | Rolled out in production |  |  |  |

### Risks & Blockers

| Risks & Blockers | Mitigation Approach |
|------------------|---------------------|
| **_New!_** |  |

### Results/Challenges/Learnings

_List any type of deliverable, e.g. merged MRs, alignment on solution, copy/designs were completed._


1.

**FYI** TAG FOLKS

Demos

Some work can take a few milestones to complete. Together with regular async issue updates, it is useful to set demos. Advantages are:

  • shorten the feedback cycle
  • increase visibility of the work
  • improve communication of work and progress

Note that providing demos and recordings is already encouraged to ease the review process in MRs.

For this purpose, a YouTube playlist has been created: Fulfillment Demos (some content is internal only). To upload your demos, you can follow the process described here. Once the video is created, link it to the Fulfillment Demos playlist. Consider adding the following info in the description:

  • A link to the Epic/Issue
  • A link to the MR (if any)
  • The name and link to profile of who uploaded it (optional)

Here’s an example (internal only)

Quality

GitLab’s Quality is everyone’s responsibility. The SETs embedded within Fulfillment section follow the Quality Engineering Department’s principles primarily to ensure that everyone is aware of product quality.

End-to-end tests - how, when and why to write them

End-to-end tests (often referred to as e2e tests) cover full or partial flows that the end-user will go through. Detailed information on testing levels can be found here.

Some examples of these flows:

  • Purchasing a SaaS subscription for a namespace on GitLab.com.
  • Purchasing a new subscription for a self-managed GitLab instance.

Since these tests are not fast and more prone to being flaky, prioritization of what to cover with end-to-end tests and what to delegate to lower level tests must be considered.

Fulfillment tests reside both in the GitLab Project and the CustomersDot Project, use the following guide to determine which project to use when writing the tests:

  • Choose CustomersDot if the test will require opening and performing actions in the CustomersDot portal, otherwise GitLab will be the correct project.
  • In the future, it is planned that CustomersDot will become just a backend service and UI end-to-end tests will reside only in GitLab.

GitLab

The GitLab end-to-end testing guide can be found here

Planned & automated Fulfillment test cases in the GitLab project can be found here.

CustomersDot

The CustomersDot end-to-end beginners guide can be found here.

Planned/automated test cases in the CustomersDot project can be found here.

Testing

The CustomersDot has different types of tests running:

  1. Linting and rubocop jobs
  2. Unit tests (specs, these could be of many types, such as controller specs)
  3. Integration tests (specs, mocking external calls)
  4. Frontend tests
  5. E2E integration tests via Watir

We also have a flag VCR that mocks external calls to Zuora by default. We have a daily pipeline that runs at 9AM UTC with the flag set so the API calls hit the Zuora sandbox and we are notified of any failure (due to potential API changes).

Any test failure is notified to #s_fulfillment_status including a link to the pipeline. Pipeline failures will prevent deployments to staging and production.

Security

Access review

Fulfillment Engineering Engineering Managers and Senior Leadership are responsible for the Fulfillment system (e.g., CustomersDot) access review on a quarterly basis. We use the following guidance to perform access reviews, access for systems will be reviewed based on the job roles and departments. As such, reviewers use their best judgement when evaluating the list of team members who currently have access while evaluating their current roles and department associations.

The following list contains some of the standard departments and teams who should retain access:

Read-only Access

  • Sales Team

Write Access

  • AppSec Team
  • Billing Team
  • Fulfillment Sub-department (engineering, product, quality, and other counterparts)
  • IT Helpdesk
  • Support Team

When in doubt, err on the side of rejecting access as it can be easily restored through another access request.

Deployment

Revenue impacting changes

Fulfillment is unique in Engineering because our changes can directly impact revenue. Changes which impact revenue or are otherwise high risk may require broader scrutiny including input from teams outside of Fulfillment. PM is the DRI on these decisions, taking input from stakeholders including Sales, Enterprise Applications, Marketing, Finance, and Customer Support.

We aspire to be as iterative as possible including even when a change potentially poses high risk.

Below are examples of changes that can be high risk because they can require coordination across many teams:

  • Pricing changes
  • Billing changes
  • Launching a new paid feature
  • Deprecating an existing paid feature
  • Changes to terms of service and related contract changes
  • Changes to the way the consumption of resources is calculated or displayed (e.g., user counts, compute minutes)

Below are changes that can often be low risk:

  • Backend enhancements that are not yet used by the frontend
  • Frontend and backend Enhancements that are behind a feature flag
  • Frontend enhancements that are not behind a feature flag, but are labeled as “beta” or “experimental”

The following processes should be followed for high-risk changes:

  • Update feature documentation of the new/updated behavior and share with relevant stakeholders
  • Use confidential issues for these changes
  • Release changes under a feature flag so this can be enabled on the release date
  • Create a rollout issue for the release date and make sure all parties are informed about the process. Example
  • Consider enabling this to a subset of customers so changes can be tested iteratively
  • Use the regular MR process even if the issue is confidential until confidential MRs are improved (1, 2)

CustomersDot

We use CD (Continuous Deployment) for CustomersDot and an MR goes through the following stages once it gets merged into the staging branch:

graph TD;
    A(Merged) --> |Green tests| B(Staging);
    B --> C[E2E Test on Staging in 40 minutes];
    C --> D[Verification]
    D --> E(Auto deploy to production in 3 hours);

If something goes wrong at the Verification stage, we could create an issue with the label production::blocker, which will prevent deployment to production. The issue cannot be confidential.

When a production blocker is in place and an MR with a fix is needed in order to remove it, the production deployment can be expedited for the MR. The job to deploy to production needs to be unscheduled to skip the 3 hour delay and triggered manually to deploy right after removing the production::blocker label or closing the issue.

For MRs with significant changes, we should consider using feature flags or create an issue with the production::blocker label to pause deployment and allow for longer testing.

Feature freeze

The feature freeze for Fulfillment occurs at the same time as the rest of the company, normally around the Friday the milestone ends.

App Feature freeze (*) Milestone ends
GitLab.com From the Friday the milestone ends to release day Same as the freeze
Customers/Provision From the Friday the milestone ends to release day Same as the freeze

(*) feature freeze may vary according to the auto-deploy document.

Any issues not merged on the current milestone post feature freeze, will need to be moved to the next one (priority may also change for those).

Production Change Lock (PCL)

There are times when GitLab temporarily halts production changes during certain events such as major global holidays, and other times where GitLab Team Member availability is substantially reduced. More information about PCLs can be found in this infrastructure handbook page.

During these PCLs, most notably at the end of the year, PCLs are managed in CustomersDot by creating an issue using the PCL template. This issue should include a checklist of instructions along with DRIs and target times for adding or removing the production::blocker label. Once the PCL has ended, the issue can be closed.

Incident management

On going Fulfillment Platform work will provide greater observability to all of the Fulfillment systems. In the meantime, however, we have some tools like CustomersDot health that post to #s_fulfillment_status in Slack. On occasion, there are reports of systemic critical trouble like a liveliness probe check failure, invalid SSL certificate, or other application errors. Look to the following sub-sections below for ways that you can contribute to error resolution.

See this handbook page for more information on GitLab Monitoring and Incident Management.

Escalation process for incidents or outages

Temporarily, while pagerslack or pagerduty is not adopted in Fulfillment, the following process is in place:

  1. When an outage occurs, the SRE on-call is notified automatically. Incidents can also be reported manually.
  2. Simultaneously to an outage, #s_fulfillment_status on Slack is notified and James Lopez and Vitaly Slobodin are paged on the phone
  3. When required, the SRE on-call or incident reporter can ping @fulfillment-engineering on Slack to notify and get help from the team

Escalation process for urgent issues

In most cases an MR should follow the standard process of review, maintainer review, merge, and deployment as outlined above. When production is broken:

  1. First determine whether the Rapid Engineering Response process should be followed. This will depend on availability and how critical the situation is.
  2. The three hour wait between Staging and Production auto deploy can be bypassed with a manual deployment by a Maintainer.
  3. When there are no project maintainers available (CustomersDot) an additional GitLab Team Member with Maintainer access can be asked to assist.

In these cases please ensure:

  1. There is an issue describing the reasons for the escalation, as per the Rapid Engineering Response. Consider ‘@‘mentioning the relevant Growth/Fulfillment teams.
  2. The change is announced on the #s_fulfillment channel.

Investigation

  • You can visit our health-check instance here. You can find the login credentials in the Subscription portal vault in 1Password. You will see services for both CustomersDot production and staging.
  • Exceptions for CustomersDot are captured in Sentry.
  • You can query logs in Kibana / Elasticsearch. Use this link to read more on using kibana at GitLab)
  • Grafana has some triage dashboards that are generally applicable for GitLab, but do not contain specific observability metrics for Fulfillment systems until the InfraDev work is complete.
  • Blackbox probes reporting production availability alerts are reported to #production in Slack

Declaring incidents

Raising an incident is always an option if you need immediate attention on a blocking problem.

Examples of blocking problems include:

  • Expired certificate causing interrupted service with staging or test environments

Critical problems like a production outage should be raised quickly. You can check #incident_management before raising an incident.

  • CustomersDot outage
  • Failed CustomersDot deploy

Grafana dashboards

These dashboards are designed to give an insight, to everyone working in a feature category, into how our code operates at GitLab.com scale.

Customer escalations

If there is a licensing issue (from support or sales) that requires escalation to product management or engineering, please create an issue via the process documented on the support handbook page under Assistance with License Issue. Please do this instead of escalating in Slack or other methods.

Fulfillment leaders in product management and engineering will subscribe to the License Issue High ARR label so they can be aware of them when they are created. This can be done in via a label search and then clicking on ‘subscribe’ for the `License Issue High ARR’ label.

The Program Manager, Cloud Licensing (currently Wayne Haber) will be primarily responsible for making the e-group aware in #e-group of licensing issues that may have a high ARR impact. The backup for the Program Manager, Cloud Licensing will be the Group Product Manager, Fulfillment (currently Jerome Ng).

Retrospectives

Monthly retrospectives

After the 8th, the Fulfillment team conducts an asynchronous retrospective. You can find current and past retrospectives for Fulfillment in https://gitlab.com/gl-retrospectives/fulfillment/issues/.

Retrospective Contributions

Everyone is encouraged to participate in asynchronous retrospective issues. This is a safe space for team member feedback. You are welcome, however, to bring your comments and concerns to the synchronous retrospective discussion or share them with your manager or other department leadership as well. We follow this guidance to produce an efficient and safe retrospective.

Some things to keep in mind:

  • Items shared do not have to be monumental successes or failures. Consider including even small items in the “what went well” and “what didn’t go so well” sections of the retrospective issue.
  • Use a team specific thread to share items that may be feel less relevant to the entire department.
  • Pull from the “wins” or “challenges” team sync agenda items for inspiration.
  • Find the limits of your computer, and keep the current retrospective issue open throughout the milestone.
Asynchronous Retrospective Synchronous Discussion

Approximately one week after the commentary period closes (on the 26th of the following month) for the async retrospective, we host a synchronous meeting to discuss next steps, action items, and improvements to our processes. This meeting can be facilitated by EMs and ICs alike. We attempt to be timezone inclusive by scheduling the meeting on a rotation between times that are APAC, AMER, and EMEA friendly. This meeting is recorded and shared for the benefit of everyone in Fulfillment.

Iteration retrospectives

There are several resources at GitLab for improving our iteration capability:

Agile software development practices uses the phrase Iteration Retrospective to describe what we perform during our Milestone Retrospective. While the Milestone Retrospective is essential in understanding how we can improve, it does have a broad focus across multiple teams. It’s a safe space where everyone can contribute through reflection on what we can continue, what we can correct, and what we can improve.

An Iteration Retrospective, in the GitLab values sense that we’re using here, is more focused on a specific example (or few examples) of a successful or unsuccessful iteration attempt. That is, an Issue, Epic, or merge request that we can examine deeply to understand what we can do differently in the future to be more iterative and efficient. These retrospectives will provide a growth opportunity to work on developing an iterative mindset - build it, then build it better.

Each group should perform their own Iteration Retrospective, but share as transparently as possible to help fellow teammates and other teams learn from your investigation.

Frequency

Iteration Retrospectives should be conducted quarterly.

Preparation

Engineering Manager(s) or Other team members should review current or previous milestones to find a good, challenging candidate issue for the Iteration Retrospective. Choose one (or two if related) from the suggestions, then create a retrospective issue.

Review these rules for an effective retrospective.

Participation

Everyone can and should contribute asynchronously to the discussion. Present your ideas on alternative methods or mechanisms to break down the problem stated in chosen issue for review. If the problem breakdown was fine, suggestions about process or other refinements are also welcome.

Don’t belabor the point, participation should be timeboxed to 1 week. A 1 week due date quick action is available in the template below.

Consider handbook updates, process changes, and bubbling up information from your conclusions after this retrospective.

Template (optional)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
Iteration is one of six GitLab Values, but also really difficult. By focusing how we have iterated well in the past and how we have not will help us iterate faster. Please contribute the following [Iteration Retrospective](#link-to-handbook-page).

## Summary

<!--
Include a brief summary of the issue. Consider including key outcomes, significant changes, and impact.
-->

`Summary of the Issue`

## Details

<!--
Complete the following details when creating the issue.
-->

- Did the team meet the iteration goals? Why or why not?
- How many MRs were created to complete the Iteration?
- What were the Days to Merge for the MRs related to the this issue?
- Are there successes or opportunities for improvement with respect to collaboration with peers or stable counterparts?
- Could the original issue have been broken down into smaller components?
- Could the MR(s) have been broken down into smaller components?

## Tasks

**Read**

- [Iteration Value](/handbook/values/#iteration)
- [Engineering Iteration](/handbook/engineering/workflow/iteration/)
- [Why iteration helps increase the merge request rate](https://about.gitlab.com/blog/2020/05/06/observations-on-how-to-iterate-faster/)

**Watch**

- [Interview about iteration in engineering with Christopher and Sid](https://www.youtube.com/watch?v=tPTweQlBS54)
- [Iteration Office Hours with CEO](https://www.youtube.com/watch?v=liI2RKqh-KA)


**Contribute**

In the threads below consider the following:

- Why was it (un)successful? If successful, how did it meet the definition of an [MVC](/handbook/product/product-principles/#the-minimal-viable-change-mvc)?
- How did the example not meet the definition of an [MVC](/handbook/product/product-principles/#the-minimal-viable-change-mvc)? In what ways could you have iterated differently? List specific examples.
- Identify areas of improvement for the team that can be incorporated into our processes and workflows.

Follow [these rules](/handbook/engineering/management/group-retrospectives/) for an effective retrospective.

Check your name off the completed list when all tasks are complete.

**Completed**

<!--
When creating this issue replace the person placeholders with teamembers and select stable counterparts as you see fit.
-->

- [ ] Person
- [ ] Person
- [ ] Person
- [ ] Person

**Next Steps**

- [ ] Engineering Manager to update your team's handbook page with ideas and improvements you've incorporated to improve iteration.

## Thread Prompts

<!--
Use these prompts or similar to start conversation threads.
-->

1. What aspects of the Issue/Epic and resulting MR(s) were very good/bad examples of Iteration and why?
1. Does anyone have alternative ideas for how this work could have been broken down?
1. Is there anything that we can change in our processes or the way we work that could improve our iteration as a team?


/due in 1 week
/label ~"Iteration Retrospective"

Project Cadence

This table lists recurring activities that are part of our Project Management Process. Having a well defined cadence makes it simple for our team to understand what activities are happening and when they happen. The listed activities closely follows our company cadence. Use this spreadsheet to edit this markdown table.

Activity Cadence Type Teams Involved
GitLab.com Deployments Continuously Async Delivery Engineering
CustomersDot Deployments Continuously Async Fulfillment Engineering
Issue Updates Weekly Async Fulfillment Section
Fulfillment Weekly Updates Weekly Async Fulfillment PMs, EMs, QEMs, UXMs
Fulfillment Group Syncs Weekly Sync Fulfillment Utilization, Purchase, Provision, and InfraDev Groups
Fulfillment PM Syncs Weekly Sync Fulfillment PMs
Fulfillment, Growth, AI Assisted EM Syncs Weekly Sync Fulfillment, Growth, AI Assisted EMs
Fulfillment PM / EM / QEM / UXM Syncs Weekly Sync Fulfillment PMs, EMs, QEMs, UXMs
OKR Check-Ins Weekly Async Fulfillment PMs, EMs, QEMs, UXMs
Monthly Self-Managed Releases Monthly Async Delivery Engineering
Monthly Release Posts Monthly Async Product and Marketing Functions
Milestone Planning Monthly Async Fulfillment Utilization, Purchase, Provision, and InfraDev Groups
Roadmap Planning Monthly Sync Fulfillment PMs, EMs, QEMs, UXMs
Monthly Product Kickoffs Monthly Sync Product Function
Product Key Reviews Monthly Sync Fulfillment PMs, Product Leadership, E-Group
Retrospective Issue Monthly Async Fulfillment Section
Retrospective Discussion Monthly Sync Fulfillment Section
Direction Review Quarterly Async Fulfillment PMs, Product Leadership
OKR Planning Quarterly Async Fulfillment PMs, EMs, QEMs, UXMs, Staff Engineers
Board Meetings Quarterly Async Fulfillment PMs, E-Group, Board

Working Groups and Cross-Functional Initiatives

This table lists recurring activities that are part of working groups and cross-functional initiatives. Use this spreadsheet to edit this markdown table.

Activity Cadence Type Teams Involved
Purchasing Reliability Working Group Weekly Sync Fulfillment Engineering, Infrastructure, IT, CEO, CoST
GTM Product Usage Data Working Group Weekly Sync Fulfillment PMs, Analytics Instrumentation, Data, Customer Success, Sales
GitLab Order to Cash Technical Fusion Team Weekly Sync Fulfillment PMs, Fulfillment Engineering, EntApps, Sales Systems
Data & Analytics Program for R&D Teams Every 2 Weeks Sync Fulfillment PMs, Analytics Instrumentation, Growth, Data
Cloud Licensing Executive Status Update Every 2 Weeks Sync Cloud Licensing Program Manager, E-Group
Ramps / Orders Bi-Weekly Every 2 Weeks Sync Purchase, Provision, EntApps, Channel Ops, Field Ops, Support, Finance, Legal
Product ARR Drivers Sync Monthly Sync Customer Success, Sales, Product Leadership
Distributor E-Marketplace Monthly Sync Purchase, Provision, EntApps, Channel Ops, Field Ops, Finance, Legal

Diversity, Inclusion, and Belonging

See the Fulfillment Section DIB page for more information about our DIB initiatives, including Fulfillment social events.

Performance indicators

See the Fulfillment Section Performance Indicators as well as the Centralized Engineering Dashboards.

Maintainer ratios

CustomersDot

Maintainer Ratio Dashboard

We attempt to have an ideal Engineer to Maintainer Ratio of 1:1 for both backend and frontend effort for our primary application, CustomersDot. The active list of maintainers can be found on the Engineering Projects page.

Some considerations that differ from other maintainer ratios:

  1. Fulfillment maintainer ratios are Maintainers vs Reviewers
  2. GitLab maintainer ratios are Maintainers vs Maintainers + Reviewers
  3. GitLab maintainer ratios do not count full-stack engineers properly (Fulfillment one does)

Knowledge sharing

Managers and direct reports

This table lists recurring activities between managers and their reports. Most of these activities are also listed on our Leadership and People Group pages. Use this spreadsheet to edit this markdown table.

Activity Cadence Type People Involved
1-1 Meetings Weekly Sync Manager, Report
Career Development Conversations Quarterly Sync Manager, Report
Skip-Levels Meetings Quarterly Sync Manager’s Manager, Report
Promotion Planning Annually Sync Manager, Report, Function Leadership, PeopleOps
Talent Assessment Annually Sync Manager, Report, Function Leadership, PeopleOps
Annual Compensation Reviews Annually Sync Manager, Report, Function Leadership, PeopleOps

Google groups

Google groups can be used for easily sending calendar invites to members of the group.

Group name
fulfillment-engineering

Communicating on Slack

List of channels

Channel Name Purpose
s_fulfillment Used for asking questions regarding the product, engineering, and Fulfillment processes.
s_fulfillment_fyi Used for announcements related to the Fulfillment sub-department.
s_fulfillment_engineering Used by the Fulfillment sub-department engineering team for internal communication and resolving internal engineering-related queries. Requests for pair programming sessions from other Fulfillment engineers can be made here.
s_fulfillment_daily Used to share daily standup updates.
s_fulfillment_status CustomersDot health monitoring channel.

Guidelines to ask questions in s_fulfillment slack channel

  1. The channel can be used to ask all product, engineering, and Fulfillment processes related questions.
  2. When your question has been answered, please react with the ✅ (:white_check_mark:) Slack emoji to the original question so that team members know that it has been answered and they do not need to look at it any longer.
  3. For urgent support issues, please follow the STAR escalation strategy.
  4. For internal support requests regarding licensing, subscriptions, trials, grace period extensions, or any requests involving a customer, please fill out the internal request form.
  5. For general licensing and subscription questions, check out the #support_licensing-subscription Slack channel.

Fulfillment Billing and Subscription Management Team
The Billing and Subscription Management Team in the Fulfillment Sub-department at GitLab
Fulfillment Diversity, inclusion, and belonging
Fulfillment Diversity, inclusion, and belonging initiatives and metrics
Fulfillment Platform Team
The Fulfillment Platform Team of the Fulfillment Sub-department at GitLab
Fulfillment Provision Team
The Provision Team of the Fulfillment Sub-department at GitLab
Fulfillment Utilization Team
The Utilization Team in the Fulfillment Sub-department at GitLab