Create:Source Code BE Team

The Create:Source Code BE team is responsible for all backend aspects of the product categories that fall under the Source Code group of the Create stage.

The Create:Source Code BE team focuses on GitLab’s suite of Source Code Management (SCM) tools and is responsible for all backend aspects of the product categories that fall under the Source Code group of the Create stage of the DevOps lifecycle. Our Product direction is found on the Category Direction - Source Code Management page.

We interface with the Gitaly and Code Review teams, and work closely with the Create:Source Code Frontend team. The features we work with are listed on the Features by Group Page and technical documentation can be found on the Create: Source Code Backend technical reference page.

Team Members

The following people are permanent members of the Create:Source Code BE Team:

Name Role
Andrew SmithAndrew Smith Core Team member
Ben BodenmillerBen Bodenmiller Core Team member
Dave MunchielloDave Munchiello Board Observer
George HoyemGeorge Hoyem Board Observer
George TsiolisGeorge Tsiolis Core Team member
Godfrey SullivanGodfrey Sullivan Lead Independent Director, Board of Directors
Hannes RosenöggerHannes Rosenögger Core Team member
Jacopo BeschiJacopo Beschi Core Team member
Karen BlasingKaren Blasing Board of Directors
Kyle DohertyKyle Doherty Board Observer
Marco ZilleMarco Zille Core Team member
Mark PorterMark Porter Board of Directors
Matt MullenwegMatt Mullenweg Board Observer
Matthew JacobsonMatthew Jacobson Board of Directors
Patrick RicePatrick Rice Core Team member
Randy GottfriedRandy Gottfried Advisor
Robert SchillingRobert Schilling Core Team member
Merline SaintilMerline Saintil Board of Directors
Siddharth AsthanaSiddharth Asthana Core Team member
Sue BostromSue Bostrom Board of Directors
Sunny BediSunny Bedi Board of Directors
Takuya NoguchiTakuya Noguchi Core Team member
Niklas van SchrickNiklas van Schrick Core Team member
Vitaliy KlachkovVitaliy Klachkov Core Team member

Stable Counterparts

The following members of other functional teams are our stable counterparts:

Name Role
André LuísAndré Luís Frontend Engineering Manager, Create:Source Code, Create:Code Review, Delivery & Scalability
Ash McKenzieAsh McKenzie Staff Backend Engineer, Create:Source Code
Costel MaximCostel Maxim Senior Security Engineer, Application Security, Plan (Project Management, Product Planning, Certify), Create:Source Code, Growth, Fulfillment:Purchase, Fulfillment:Provision, Fulfillment:Utilization, Systems:Gitaly
Derek FergusonDerek Ferguson Group Manager, Product Management, Create
Darva SatcherDarva Satcher Director of Engineering, Create
Gavin HinfeyGavin Hinfey Backend Engineer, Create:Source Code
Igor DrozdovIgor Drozdov Staff Backend Engineer, Create:Source Code, Systems:Gitaly API
Senior Backend EngineerSenior Backend Engineer Senior Backend Engineer, Create:Source Code
Javiera TapiaJaviera Tapia Backend Engineer, Create:Source Code
Jacques ErasmusJacques Erasmus Senior Frontend Engineer, Create:Source Code
Patrick CyizaPatrick Cyiza Backend Engineer, Create:Source Code
Joe WoodwardJoe Woodward Senior Backend Engineer, Create:Source Code
Kai ArmstrongKai Armstrong Principal Product Manager, Create:Code Review
Marcin Sędłak-JakubowskiMarcin Sędłak-Jakubowski Senior Technical Writer, Plan (Project Management, Product Planning), Create (Source Code)
Marie-Christine BabinMarie-Christine Babin Senior Product Manager, Create:Source Code
Nataliia RadinaNataliia Radina Frontend Engineer, Create:Source Code
Paulina Sedlak-JakubowskaPaulina Sedlak-Jakubowska Frontend Engineer, Create:Source Code
Robert MayRobert May Senior Backend Engineer, Create:Source Code
Shekhar PatnaikShekhar Patnaik Principal Fullstack Engineer, Create
Vasilii IakliushinVasilii Iakliushin Staff Backend Engineer, Create:Source Code, Systems:Gitaly API

Sisense & KPIs

We have a metrics dashboard to help us stay on track with Development KPIs (Make sure you filter by our team at the top!) This dashboard does not include security MRs from dev.gitlab.org, but does include security MRs from production.

Workflow

We use the standard GitLab engineering workflow. To create an issue for the Create:Source Code BE team, add these labels:

  • ~backend
  • ~“devops::create”
  • ~“Category:Source Code Management”
  • ~“group::source code”

For more urgent items, feel free to use #g_create_source_code on Slack.

Take a look at the features we support per category here.

Working with the Product Team

Weekly calls between the Product Manager and Engineering Managers (frontend and backend) are listed in the “Source Code Group” calendar. Everyone is welcome to join and these calls are used to discuss any roadblocks, concerns, status updates, deliverables, or other thoughts that impact the group.

Issue refinement

  1. Once we have validated the problem, product, UX, and engineering will collaborate to propose a solution and decide on what’s technically feasible. The proposed solution will be shared with users to validate it solves the problem.
    1. Issues that require design work are marked with UX and workflow::ready for design.
    2. Once designs are ready and the proposed solution is viable then the label workflow::planning breakdown will be applied.
  2. Once we have confirmed the proposed solution is viable, we will move to break it down as much as possible. When issues are ready for this stage, PM will mark issues with workflow::refinement label to signal next step.
  3. EM will work with engineers to decide distribution of work and assign issues for breakdown.
  4. Engineers or EM will evaluate the issue, work with PM, UX, and other engineering counterparts where necessary to address questions and concerns.
  5. If the planned implementation of the issue can be further broken down, the engineer/EM will work with the PM to reduce scope and create new issues until this is the case (either PM or engineer/EM can create new work items).
  6. When the planned implementation of the issue is in its smallest form, the engineer/EM will provide a weight. EM or PM will mark as workflow::needs issue review.

Note: if an issue receives a weight > 3 after this process, it may indicate the IC may not have a full idea of what is needed and further research is needed.

Convention over configuration

As stated in our direction, we must place special emphasis on our convention over configuration principle. As the feature set within Create:Source Code grows, it may feel natural to solve problems with configuration. To ensure this is not the case, we must intentionally challenge MVC and new feature issues to check for this. Let’s consider the following steps for best results:

  1. Once issues have been labeled as workflow::needs issue review PM will share the proposal with either a peer or their manager as well as engineering (EM or IC) and product designer.

  2. Peers in product and engineering who review the issue should look for opportunities to eliminate configuration where possible. If opportunities are identified, the issue is moved back to workflow::refinement.

  3. If PM and peers are satisfied with the proposal and it follows our convention over configuration principle as much as possible, those who reviewed the issue indicate their agreement with the proposal (with either 👍 or a comment in the issue). Finally, PM or EM will label issue workflow:: ready for development.

Collaborating with counterparts

You are encouraged to work as closely as needed with stable counterparts outside of the PM. We specifically include quality engineering and application security counterparts prior to a release kickoff and as-needed during code reviews or issue concerns.

Quality engineering is included in our workflow via the Quad Planning Process.

Application Security will be involved in our workflow at the same time that kickoff emails are sent to the team, so that they are able to review the upcoming milestone work, and notate any concerns or potential risks that we should be aware of.

Communication

As a team we strive to be responsive and accommodating when we communicate. When consuming information, we use the following emoji to definition table as a shorthand glossary:

Emoji Definition
👀 I’m looking at this now
🤔 I need to think about this before responding
📍 I don’t have time, will come back later
🤷🏼 I don’t know
👌 I understand
👍 Agree (and often won’t give additional comment)
Task is complete
I’ve seen this but I don’t think I’m the best person for the job. Ping me if no one else responds and you need help

Triage Process

The weekly Triage Report is generated automatically by the GitLab bot and this report is reviewed by the EM. Here is an example of a previous report.

The Triage Report can be quite long, and it important to deal with it efficiently. An effective way to approach it is:

  • Open every issue in a separate browser tab and use “edit issue” to mark then as checked once review, then close the tab.
  • Verify if the issue belongs to ~“group::source code” and change group label if needed. The Features by Group page is a good starting point for this assessment.
  • Apply ~frontend if it is a frontend issue.
  • Perform a brief search to assess if is a duplicate, close with a ~Duplicate label if this is the case.
  • Is it a ~“support request” ? Does it ~“needs investigation” ? Apply labels if so.
  • Apply ~regression label if it is one, consider bumping severity numbers if recent.
  • Apply ~“severity::4”, ~“priority::4”, %Backlog if a smaller issue with no significant impact.
  • If an uncontroversial problem with a clear solution, consider applying ~“Seeking community contributions”
  • If also an easier issue which might interest a newer community contributor, consider applying ~“quick win”.
  • Apply ~“priority::3” ~“severity::3” if a bug with a workaround.
  • Anything causing data loss, severe performance impact or security apply a ~“severity::1” and ~“priority::1” or ~“priority::2” and assign to a team member.
  • Unassign yourself from the Triage Report

Engineering cycle

The engineering cycle is centered around the GitLab Release Date every month. This is the only fixed date in the month, and the table below indicates how the other dates can be determined in a given month.

Iteration Documents

These documents comprise everything that is documented during the release planning and execution.

Issue Boards

Create Source Code BE planning takes inputs from the following sources:

Create Source Code UX planning takes inputs from the following sources:

Planning Issue

Each month a planning issue is created by the PM, using the Source code template.

Planning Board

The Planning Board is created for each release by the PM, and is a curated list of issues by category. The EM requests engineers to allocate weights to all issues on this board via the Needs weight issue

Capacity Planning Spreadsheet

The EM maintains a Google Sheet for calculating team capacity, and the same Spreadsheet is also used to perform the process of assigning issues to the release based on weight and priority.

Build Board

The EM selects issues from the Planning Board based on:

  • slipped issues
  • weight
  • priority
  • PM preference

The EM then applies the ~Deliverable label to each issue in the Release and assigns then to an engineer. The issues are tracked through the release via the Build Board.

Candidate issues

Urgent issues are tentatively assigned to a release to ensure other teams have visibility.

At this point the issues are Candidate issues, and the milestone does not confirm that they will be definitely scheduled. Issues move from Candidate status to confirmed during the Issue selection process.

Key Dates

Date Event
The Monday of the week the milestone ends PM creates planning board and pings EMs in the Planning Issue for review & weighting.

EMs calculate capacity, add to Planning Issue.

PM submits RPIs for reviews.
Monday to Friday of the week the milestone ends EMs & ICs add weights to issues in the planning board
The Friday the milestone ends EMs add ~Deliverable labels to issues so that they appear on the Build board as a draft

Release Post: EMs, PMs, and PDs contribute to MRs for Usability, Performance Improvements, and Bug Fixes
The Friday the milestone ends EMs adjust ~Deliverable labels for slippage and make final assignments

PMs review final plan for milestone on Build board

EMs merge RPI MRs for features that have been merged.
The third Thursday of the month Release

Weighting issues

Based on the issues on the Planning Board, the EM will create a Needs Weight issue to request an estimation of work by the engineers. In general no more than 4 issues should be assigned to an engineer for weighting.

  1. Give issue a weight if there’s none yet or update if the existing weight isn’t appropriate anymore. Optionally leave a comment describing why a certain weight is given.
  2. It’s strongly encouraged to spend no more than 1 hour per issue. Give it your best guess and move on if you run out of time.
  3. Label the issue as ~“workflow::ready for development” if you feel we can make progress on it in the next milestone.

If you would like to be assigned to work on this issue in the upcoming release, add a comment and ping the EM.

Weight categories

The weights we use are:

Weight Description
1: Trivial The problem is very well understood, no extra investigation is required, the exact solution is already known and just needs to be implemented, no surprises are expected, and no coordination with other teams or people is required.

Examples are documentation updates, simple regressions, and other bugs that have already been investigated and discussed and can be fixed with a few lines of code, or technical debt that we know exactly how to address, but just haven’t found time for yet.
2: Small The problem is well understood and a solution is outlined, but a little bit of extra investigation will probably still be required to realize the solution. Few surprises are expected, if any, and no coordination with other teams or people is required.

Examples are simple features, like a new API endpoint to expose existing data or functionality, or regular bugs or performance issues where some investigation has already taken place.
3: Medium Features that are well understood and relatively straightforward. A solution will be outlined, and most edge cases will be considered, but some extra investigation will be required to realize the solution. Some surprises are expected, and coordination with other teams or people may be required.

Bugs that are relatively poorly understood and may not yet have a suggested solution. Significant investigation will definitely be required, but the expectation is that once the problem is found, a solution should be relatively straightforward.

Examples are regular features, potentially with a backend and frontend component, or most bugs or performance issues.
4: Large Features that are well understood, but known to be hard. A solution will be outlined, and major edge cases will be considered, but extra investigation will definitely be required to realize the solution. Many surprises are expected, and coordination with other teams or people is likely required.

Bugs that are very poorly understood, and will not have a suggested solution. Significant investigation will be required, and once the problem is found, a solution may not be straightforward.

Examples are large features with a backend and frontend component, or bugs or performance issues that have seen some initial investigation but have not yet been reproduced or otherwise “figured out”.
5: Unknown A feature that is weight 5 will not be scheduled and instead should be broken down or a spike scheduled

A weight of 5 generally indicates the problem is not clear or a solution should be instead converted to an Epic with sub-issues.

If the issue should be broken down

If the problem is well-defined but too large (weight 5 or greater), either:

  • Promote the issue to an Epic and break the work into sub-issues. Weight the individual issues if possible.
  • Ping @sean_carroll and @tlinz and outline the reason the issue needed to be promoted to an Epic.
If the issue SSOT is not clear
  • Don’t assign a weight, instead add a comment indicating what needs clarification and ping @sean_carroll and @tlinz.
If the issue needs a spike
  • Don’t assign a weight, instead add a comment about the need for a spike (and possibly what would be investigated) and ping @sean_carroll / @tlinz.
  • Spikes are scheduled with a weight of 2.
  • Spikes are scheduled with a weight of 2 (timeboxed).
  • The ~spike label is added

When a spike is scheduled, the engineer performs research on what needs to be done. On completion of the investigation, the engineer has either closed the issue or developed a plan for the work needed, including a weight. A follow-up issue is created and the labels copied from the original issue, and the original issue then closed.

Security Issues

Security issues are typically weighted one level higher than they would normally appear from the table above. This is to account for additional work and backports in the security release process.

Planning issue review

The Source Code stable counterparts (BE, FE, PM, UX) meet and propose issues to be worked on in the upcoming release. Using the Mural visual collaboration tool, candidate issues are voted on by the group.

Capacity planning

Capacity planning is a collaborative effort involving all Source Code team members and stable counterparts from Frontend, UX and Product. An initial list of issues is tracked in the Source Code Group Planning issue example for each month.

Team Availability

Approximately 5-10 business days before the start of a new release, the EM will begin determining how “available” the team will be. Some of the things that will be taken into account when determining availability are:

  • Upcoming training
  • Upcoming time off / holidays
  • Upcoming on-call slots
  • Potential time spent on another teams deliverables

Availability is a percentage calculated by (work days available / work days in release) * 100.

All individual contributors start with a “weight budget” of 10, meaning they are capable (based on historical data) of completing a maximum number of issues worth 10 weight points total (IE: 2 issues which are weighted at 5 and 5, or 10 issues weighted at 1 each, etc.) Then, based on their availability percentage, weight budgets are reduced individually. For example, if you are 80% available, your weight budget becomes 8.

Product will prioritize issues based on the teams total weight budget. Our planning rotation will help assign weights to issues that product intends on prioritizing, to help gauge the amount of work prioritized versus the amount we can handle prior to a kickoff.

Source Code issue pipeline

The Source Code issue pipeline is broad, and the PM and EM work together throughout the planning process and the final list is chosen during the Issue Selection meeting. The issue pipeline includes:

Issue Selection

On or around the 16th, the PM and EM meet once more to finalize the list of issues in the release. The issue board for that release is then updated, and any issues with an candidate milestone that are not selected will be moved to Backlog or added as a candidate for a future release.

Issues scheduled for the release are then marked ~“workflow::ready for development”.

Issue assignments

Once availability has been determined, weights have been assigned, and the PM/EM finalize a list of prioritized issues for the upcoming release, kickoff emails will be sent. The intent of this email is to notify you of the work we intend to assign for the upcoming release. This email will be sent before the release begins. The kickoff email will include:

  • Your availability, weight budget, and how it was calculated
  • A list of the issues you will most probably be assigned as an individual
  • A reasoning behind why you have been assigned more than your weight budget, if applicable
  • A list of the issues the team is working on that are deemed “note-worthy,” in case you’d like to offer help on those issues as time allows

You will begin to collect follow-up issues when you’ve worked on something in a release but have tasks leftover, such as technical debt, feature flag rollouts or removals, or non-blocking work for the issue. For these, you can address them in at least 2 ways:

  • Add an appropriate future milestone to the follow-up issue(s) with a weight and good description on the importance of working this issue
  • Add the issue(s) to the relevant planning issue

You should generally take on follow-up work that is part of our definition of done, preferably in the same milestone as the original work, or the one immediately following. If this represents a substantial amount of work, bring it to your manager’s attention, as it may affect scheduling decisions.

If there are many follow-up issues, consider creating an epic.

Backend and Frontend issues

Many issues require work on both the backend and frontend, but the weight of that work may not be the same. Since an issue can only have a single weight set on it, we use scoped labels instead when this is the case: ~backend-weight::<number> and ~frontend-weight::<number>.

Workflow labels

The easiest way for engineering managers, product managers, and other stakeholders to get a high-level overview of the status of all issues in the current milestone, or all issues assigned to specific person, is through the Development issue board, which has columns for each of the workflow labels described on Engineering Workflow handbook page under Updating Issues Throughout Development.

As owners of the issues assigned to them, engineers are expected to keep the workflow labels on their issues up to date, either by manually assigning the new label, or by dragging the issue from one column on the board to the next.

Retrospectives

We have 1 regularly scheduled “Per Milestone” retrospective, and can have ad-hoc “Per Project” retrospectives.

Per Milestone

The Create: Source Code group conducts monthly retrospectives in GitLab issues. These include the backend team, plus any people from frontend, UX, and PM who have worked with that team during the release being retrospected.

These are confidential during the initial discussion, then made public in time for each month’s GitLab retrospective. For more information, see team retrospectives.

Per Project

If a particular issue, feature, or other sort of project turns into a particularly useful learning experience, we may hold a synchronous or asynchronous retrospective to learn from it. If you feel like something you’re working on deserves a retrospective:

  1. Create an issue explaining why you want to have a retrospective and indicate whether this should be synchronous or asynchronous
  2. Include your EM and anyone else who should be involved (PM, counterparts, etc)
  3. Coordinate a synchronous meeting if applicable

All feedback from the retrospective should ultimately end up in the issue for reference purposes.

Deep Dives

The groups in the Create stage organize regular Deep Dive sessions to share our domain specific knowledge with others in the stage, the organization, and the wider community. All existing Deep Dives can be found on GitLab Unfiltered with slides and recordings in the descriptions. For Create specific Deep Dives, there is a different playlist. To find more information on upcoming sessions, or to propose a new topic, please see the epic.

Career development

Career development conversations in the Create:Source CodeBE team are centered around a Career Development Sheet that is based on the Engineering Career Matrix for Individual Contributors. The sheet lists the expected current level behaviors on the left, the next level behaviors on the right, and uses colored columns in between to visually represent the extent to which the individual has shown to have grown from the current level to the next. Columns to the right of a next level behavior are used to collect specific examples of that behavior, which serve as evidence of the individual’s growth.

Both the matrix and the sheet are Works in Progress; the development of the career matrix is tracked in an epic, and as the matrix evolves, the sheet will be updated to match.

Performance Monitoring

The Create:Source Code BE team is responsible for keeping some API endpoints and controller actions performant (e.g. below our target speed index).

Here are some Kibana visualizations that give a quick overview on how they perform:

These tables are filtered by the endpoints and controller actions that the group handles and sorted by P90 (slowest first) for the last 7 days by default.

Last modified February 26, 2024: update chart links on Dev group pages (8407fe98)