The Utilization Team often works at the interface between GitLab Core and Fulfillment applications. This includes components in the Utilization category like consumables management (storage, compute, seats, etc.), usage reporting, and usage notifications. Our customers use GitLab SaaS, GitLab self-managed, and internal tooling.
For more details about the product vision for Fulfillment, see our Fulfillment page.
The Utilization group manages the Utilization category.
| Person | Role |
|---|---|
| Costel Maxim | Senior Security Engineer, Application Security, Plan (Project Management, Product Planning, Certify), Create:Source Code, Growth, Fulfillment:Purchase, Fulfillment:Provision, Fulfillment:Utilization, Systems:Gitaly |
| Jerome Ng | Director of Engineering, Fulfillment, Interim Strategy and Operations Lead, JiHu |
| Nick Malcolm | Senior Security Engineer, Application Security, Data Science (AI Assisted, Anti-Abuse, MLOps, DataOps), Fulfillment:Utilization, Incubation Engineering |
| Ottilia Westerlund | Security Engineer, Fulfillment (Fulfillment Platform, Billing and Subscription Management), Govern (Security Policies, Threat Insights), Monitor (Observability), Plan (Product Planning) |
| Vincy Wilson | Senior Manager, Quality Engineering, Enablement, Fulfillment, Growth, Sec and Data Science |
| Omar Fernandez | Director of Product Management, Fulfillment |
We try to work async as much as possible. However, there are occasions where synchronous communication might be better suited for the task. Our Weekly Team Sync meeting is one of those occasions where we can celebrate wins, collaborate quickly over a number of issues, and generally have some face-time in an all-remote environment.
Crucially, we use this high bandwidth communication to review issues that need some extra discussion and input from the team to ensure that issues can be estimated and work developed with minimal back and forth to establish the nature of the request.
The Utilization group is spread across at least 4 different timezones.
The Utilization Team meets via Zoom on Tuesdays each week at 9:00 AM US Eastern Time (6:00 AM PT / 2:00 PM UTC) to accommodate teammates in their timezones.
Using the [REC] meeting prefix, the meeting is automatically uploaded into the GitLab Videos Recorded folder in Google Drive on an hourly basis via a scheduled pipeline. All teammates are set as alternate hosts and should be able to record the meeting if the Engineering Manager is not present. The recording link will be placed into the agenda after the recording is available.
We don't normally publish our team sync meetings to the GitLab Unfiltered YouTube channel because we often talk about confidential issues related to product pricing and other internal initiatives. We would rather have a single repository of meeting recordings in Zoom or Google Drive rather than risk of unsafe information disclosure.
All team members are encouraged to add topics to the weekly agenda async. In the unlikely event that the agenda is empty, we'll cancel the meeting.
Engineers are responsible for providing async issue updates on active, assigned issues when progress is made. Following the template and guidance for async updates for the entire Fulfillment Sub-department, updates should be made at least weekly. These updates help keep collaborators, stakeholders, and community contributors informed on the progress of an issue.
The Engineering Manager will report before the end of each week on milestone progress in the current milestone planning issue on the following topics:
**Total Weight Closed** XX
**Total Weight Open** XX (XX in dev)
**Deliverable Weight Closed** XX
**Deliverable Weight Open** X (XX in dev; X blocked)
**Blocked Issues** X issue(s) (X weight) link, description
The Engineering Manager will report on the progress of OKRs every two weeks as a comment in relevant work items found in the Objectives and Key Results project.
Current OKRs: Work Items Labeled with Utilization
It is important to take time off so that you can rest, reset, avoid burnout, take care of loved ones, etc. You are encouraged to take time for yourself or your family following our operating principle of Family and Friends First, work second. Guidance is to communicate your PTO intention with your manager with twice the number of days that you are taking off.
When going out of office, please be sure to clearly communicate your availability with other people. The following steps are required when submitting a PTO notification.
In Time Off by Deel, select a role as your backup during your PTO. Please assign the team slack channel #g_utilization as your backup to help distribute the workload. Consider if your current work in progress requires a substitute DRI and assign a single person for those specific issues.
Add the Fulfillment Shared Calendar to your Time Off by Deel settings so your PTO events are visible to everyone in the team. The calendar ID is: gitlab.com_7199q584haas4tgeuk9qnd48nc@group.calendar.google.com Read more about PTO in the handbook.
Update your GitLab.com status with your out of office dates by clicking on your profile picture and selecting "Edit Status." For Example: OOO Back on yyyy-mm-dd Adding OOO to your status message will keep you from appearing in the reviewer roulette.
| Date | Event |
|---|---|
| 10th | PM creates a Planning Issue and pings the EM(s) in the Planning Issue for review & preliminary weighting. EM and PM calculate capacity, add to Planning Issue. |
| 10th-14th | EM & ICs add weights to issues in the backend and frontend build boards. |
| 15th | PM adds ~Deliverable labels to issues. |
| 17th | Last day of milestone PM adjusts current and upcoming issues to reflect slippage from the current milestone. ~Deliverable labels are adjusted as necessary. |
| Team Sync Closest to the Next Milestone | PM reviews the upcoming milestone plan with the team. |
| 18th | Milestone begins |
| 22nd | Release |
We have cross-functional prioritization process that is aligned with our prioritization framework.
type::feature issuestype::maintenance issuestype::bug issuessus::impacting issues (System Usability Scale (SUS))Before work can begin on most issues, a preliminary investigation is often required to provide an estimate and breakdown of the work as described. If the scope of work for a given issue spans several disciplines (docs, design, frontend, backend, etc.) and involves significant complexity across those disciplines, consider creating separate issues for each discipline.
Engineers are expected to allocate approximately 4 hours each milestone to evaluate, refine, and weigh issues assigned to them. Refining an issue ensures the problem statement is clear enough and designs are available (if applicable) to provide a rough effort sizing estimate; the intention is not to provide solution validation during refinement.
If any issue requires any additional ~frontend, ~backend, ~Quality, ~UX, or ~documentation reviews, they are assigned to the respective individual(s) to collaborate on the issue where they can validate assumptions and/or provide missing detail necessary to provide an estimate.
Anyone on the team can contribute to answering the questions in this checklist, but the final decisions are up to the PM and EM.
### Refinement / Weighting
<!--
Ready for development means replying yes to the following questions:
- Does the issue have a problem statement in the description?
- Does the issue have the expected behavior described well enough for anyone to understand?
- Does the issue explicitly define who the stakeholders are (e.g. BE, FE, PM, UX and/or Tech Writer)?
- Does the issue have a proposal in the description?
- Does the issue have proper labeling matching the job to be done? (e.g. bug, feature, performance)
- Is this issue sufficiently small enough? If not, break it into smaller issues
- Is it assigned to the correct domain (e.g. frontend, backend)? If not, break it into two issues for the respective domains
- Is the issue clear and easy to understand? If not, try asking further clarification questions and update the description once they are received
- Has the issue been reviewed for overlap with other issues and addressed as necessary?
- Are the weights of the issues updated based on team input and knowledge?
If more than 2 MRs are needed, consider adding a table like the following to the description (e.g. under `Implementation plan`).
| Description | MR |
|-|-|
|||
It will help track the status.
-->
- [ ] Ready for development
- [ ] Weight is assigned
- [ ] Number of MRs listed
- [ ] Needs testing considerations
- [ ] Needs documentation updates
- [ ] Reviewed other milestone issues for possible overlap with current one and mark them as "linked", "blocks", or "is blocked by"
**Reasoning:**
<!--
Add some initial thoughts on how you might break down this issue. A bulleted list is fine.
This will likely require the code changes similar to the following:
- replace the hex driver with a sonic screwdriver
- rewrite backups to magnetic tape
- send up semaphore flags to warn others
Links to previous examples. Discussions on prior art. Notice examples of the simplicity/complexity in the proposed designs.
-->
needs weight and workflow::planning breakdown labels. This board tracks those assignments.~workflow::* label to the appropriate status, e.g.
~"workflow::design" if further design refinement is needed, and let the designer know.~"workflow::ready for development" when refinement is completed and a weight has been applied, signaling that it's ready for implementation and the issue can now be prioritized.~"workflow::planning breakdown" if extensive investigation and/or research is needed, the status does not move, and the PM and EMs should be informed. An issue remains in this workflow status while collaborating with others to clarify minor aspects of the issue.If an issue has several components (e.g. ~frontend, ~backend, or ~documentation) it may be useful to split it up into separate implementation issues. The original issue should hold all the discussion around the feature, with separate implementation issues being used across disciplines to track the work done. Doing this provides several benefits:
Sometimes a long passage of time might elapse between the initial refinement and an engineer picking up the issue to work on it. With the frequency with which the codebase can be updated, this can lead to the implementation plan or the weight being inaccurate and in some cases the issue might not even be required anymore.
If that passage of time is significant, e.g. 1 year, we should re-evaluate the issue to ensure it is still relevant and accurate by performing the refinement process, using the Refinement/Estimation Template above.
We have many labels that can be applied to an issue or a merge request. Besides the issue workflow labels, here are the minimum basic labels to apply to issues and merge requests:
type::feature, type::bug, or type::maintenance)section::fulfillment)group::utilization)frontend, backend, database, UX, documentation)security if the issue is related to application security, and breaking change if this work is considered a breaking changeThe workflow::planning breakdown label is driven by Product, but is a collaborative effort between Product, UX and Engineering. We are using this label to help drive the refinement, estimation, and breakdown of issues as described above. During that refinement process, the following examples of how the team might contribute towards that effort:
~UX label.Spikes are labeled with spike, feature::enhancement and type::feature. Spikes that update dependencies or upgrade versions of underlying libraries are labeled with spike, maintenance::dependency and type::maintenance
We follow the estimation rubric provided on the Fulfillment handbook page that equates issue weight mostly with complexity and breadth of change required.
As a general rule, issues with an estimated weight above 5 should be broken down into smaller, more iterative issues. The Engineer assigned to the issue for estimation will suggest how to split the issue. If the Engineer has a clear understanding about how the split should happen, they should proactively split the issue while also leaving estimations on the child issues along with a comment on the parent issue. If the split results is more than 4 issues, consider creating an epic.
Estimations should help us:
Estimations are not perfect, nor should we spend many hours trying to provide the most perfect estimation. As a result, we do not expect:
Sometimes during development, we reshape the issue, change its solution, or even postpone it. We are proud to work to contribute to issues even if the outcome shifts from our original notion.
Estimation Review
We do not have a synchronous meeting to discuss issue estimates like in a planning poker session where estimations are blindly chosen and shared only to reach a consensus from the entire group; this activity does make estimations more engaging and accurate than other methods. Like other GitLab activities, we attempt to do this asynchronously, but limited to a few team members.
In an effort to raise our confidence of our estimates through shared discussion, ask another team member to briefly review the issue and your rationale for estimation, but conceal the weight of your estimate in a details block (see below). Should both team members agree, or be moved to either side, we will use that estimate. If there is some disagreement, about the estimate value provided as a result of new information, experience, or opinion, we have options to choose either value, an average value that fits within the estimation framework, or seeking a third party opinion like that of the engineering manager.
<details>
<summary>Estimate</summary>
estimate: 2
</details>
Throughout the estimation process, consider the following GitLab sub-values:
We use spikes during the milestone to produce a design document or similar artifact that is useful for planning and execution of feature or other complex work. This can simply be, but not limited to, an issue containing a summary of the discussions on the topic, answers to questions from the spike description, links to any Proof of Concept MRs produced, a road map or other detailed outline. They are assigned like other work and take up capacity from the milestone.
Spikes are typically considered a Deliverable issue and we use this template (internal only) to create new spike issues.
Estimating the effort required for a Spike is not as clearly set or easily defined as feature work because the complexity can't be accurately estimated. Take into consideration the following criteria when adding a weight to spike issue.
Follow the same Fibonacci scale used for feature work from 1 (low, quick, easy) to 5 (large, lengthy, difficult). Historically, we have not estimated spikes higher than 5.
Guidelines
P1/S1 bug type issues, they will outrank other planned issues.Deliverable and group::utilization.Deliverable issues have been picked up as denoted by assignment and are in the workflow:in dev stage or beyond, then you should pull from the current milestone column on the respective boards. The current milestone column will be stack ranked (or at least reviewed) daily by the Product Manager, so that each team member can pull from the top of the column expecting that it is already ordered in priority.Stuff that should just work1, unscheduled work, or other areas of interest.DRIs assign themselves to the selected issue. This is also a good time to re-evaluate the weight and proposal, in case the DRI picking up the issue was not the same individual who originally weighted and refined the issue. Aspirationally, we strive to iterate and want to breakdown issues to ship as much value in the milestone for our users as possible, which means if you see a more efficient way forward when you start working on a new issue, you are free to raise a comment and update the proposal to deliver more iterative value.
The following lists are links to Sentry and other tools where we proactively identify and triage Utilization problems. Proactive triage will not only provide for a more secure and robust application, but also provide for a better user experience especially as we iterate on features, reveal features from behind a feature flag, or introduce a refactoring. It leans into our Bias for action sub-value and raises our awareness of application performance.
| Subject | Link | Notes |
|---|---|---|
| CustomersDot syncing | Sentry | UpdateGitlabPlanInfoWorker class is used to sync data between CustomersDot and GitLab |
| GitLab Subscriptions | Sentry | Results could be refined by controller, e.g. SubscriptionsController |
| Billing errors | Sentry | Results could be further refined by controller, e.g. Groups::BillingsController, Projects::BillingsController |
| Rails logs | Kibana | Utilization feature category Rails logs for the last 7 days |
| Sidekiq logs | Kibana | Utilization feature category Sidekiq logs for the last 7 days |
| Billable Member API | Grafana dashboard | - |
| CustomersDot Bug Issues | Issues | - |
| GitLab Bug Issues | Issues | - |
There's a way in sentry to create an issue for any error you find.
e.g. https://sentry.gitlab.net/gitlab/customersgitlabcom/issues/2505559/?query=is%3Aunresolved%20UpdateGitlabPlanInfoWorker
See links in the right sidebar:
Although both links look the same, the first link is for creating an issue in the security repo, the second should be for the project (CustomersDot/GitLab) accordingly.
Fulfillment has a new collaborative process with the Support team to track and manage requests for help. We have a specialized, scoped label, ~"Help group::Utilization". Issues are generated in the same project.
Other boards where previous requests are tracked:
Iteration powers Efficiency and is the key that unlocks Results, but it's also really difficult to internalize and practice. Following the development department's key metrics we strive to deliver small and focused MRs.
Following a similar process to Milestone Retrospectives, we employ Iteration Retrospectives on a quarterly basis.
Key Takeaways
Each month, we conduct an architecture review.
The review process targets reviews from both a current and future design perspective. That is, the group decides if the component under review should be done in a particular way. This is not necessarily "should a thing be done," but it could be used that way. It can also be used to propose a large architectural change that might have wide sweeping changes or as a prelude to the construction of a blueprint in our evolution workflow process.
The timeline has both async and sync steps. For smaller reviews, it may be possible to complete all of the steps asynchronously. We've already established that synchronous time is valuable. This process follows a similar pattern that we have used for our retrospectives and working groups.
/doc directory of the CustomersDot projectThe time required to prepare, discuss, and complete a review is variable. We should not spend all of our time on the review. We should approach the review pragmatically with a focus on understanding the problem, the value proposition, and definition of success. By meeting on a 30 day cycle, we can spread out the investment required over time.
We should time box the preparation of the review, and we should reconsider or reduce the review scope if the topic is too broad, has too many hidden dependencies, or becomes unwieldy in some other way.
Team collaboration is important to the success of the review. The Reviewer should ask for input, feedback, and receive help from other team members by opening a discussion thread in the review issue. The role is not meant to be solitary, but a typical DRI.
(Sisense↗) We also track our backlog of issues, including past due security and infradev issues, and total open System Usability Scale (SUS) impacting issues and bugs.
(Sisense↗) MR Type labels help us report what we're working on to industry analysts in a way that's consistent across the engineering department. The dashboard below shows the trend of MR Types over time and a list of merged MRs.
(Sisense↗) Flaky test are problematic for many reasons.
(Sisense↗)
(Sisense↗)
Small effort improvements to the project can be added to the current milestone as unplanned work. Resolving small bugs or fixing flakey tests immediately are examples of how to use the stuff that should just work label. ↩