Scale and develop our diverse, global team to drive results that support our product and customer growth, while maintaining our values and unique way of working.
GitLab's unique way of working asynchronously, handbook first, using the product we develop, and with clear focus on our values enables very high productivity. In delivering on growth, we maintain our values and ways of working while developing team members and increasing the diversity of our team. We focus on constantly improving usability and reliability of our product to reach maximum customer satisfaction. Community contributions and customer interactions rely on efficient and effective communication. We are a data-driven, customer experience first, open core organization delivering one secure, reliable, world leading DevOps platform.
FY23 has similar ambitions as FY22. During FY22, the Development department focused on reliability/security, team, SUS, and product. We added EA (Engineering Allocation) and the FCL (Feature Change Lock) processes to support the improvements in reliability and security. We reduced the number of past due issues in both categories by 77% and 72% respectively, and continue focusing on reducing this further. We have grown the team by 22% during FY22 while retaining 90% of team members, and have set ourselves up for further growth in FY23. We have begun investing further in SUS improvements after seeing a 7% improvement via various activities, including component migrations. We have improved the product using standard product management and development processes. Users want to see these improvements as well as improvements in reliability and security.
As we have set ourselves at the end of FY22, FY23 will focus on growth, efficiency, usability, product, and diversity.
This year we have goals to increase the size of the Development Department by over 20% to increase our team's capacity to deliver in new and existing product areas. To achieve this, we are focused on collaborating closely with our recruiting team, building a diverse pipeline of candidates, and successfully onboarding new team members into our way of working. Each sub-department will set goals for their teams to hire based on planned headcount increases. We expect to meet our hiring goals while not going over, which is essential to 1) meeting the growing development needs of GitLab's business strategy and 2) hiring at a predictable rate in-line with our financial plans. We commit to investing dedicated time and budget in learning and development, specifically to development topics through internal courses, such as O'Reilly Learn, etc.
GitLab's Development group ships thousands of product merge requests per month. Continuing to scale our development process to an ever larger number of contributors requires efficiency, collaboration, and iteration. In FY23, we want to keep our MR Rate stable as we continue to onboard new team members, increasing our overall output and demonstrating the scalability of our approach to development. We will focus on training our new hires on iteration and process improvements, saving team members time. We will also review the best metrics to focus on and are considering moving back to an overall MR Rate measure (from an authorship one). Doing so will help us measure the efficiency of our responsiveness to our peers for the company and the community.
User experience is a continued focus area for FY23. Millions of customers use GitLab so UX improvements can have a huge collective impact across all of these individuals. We support this effort both in the product development as well as in our architecture. This includes continued conversion of Pajamas components in order to continue to improve performance experienced by users. Also efforts like burning down significant (Sev 1 and Sev 2) usability issues, addressing Usability Benchmark insights, are part of our short and long term goals.
Development team members should also constantly suggest and investigate how to improve the overall user experience of the product. These can range from enhancing performance (actual and perceived), suggesting new technologies, solving user experience issues efficiently, etc.
Lastly, we will continue our strong partnership with Product to make GitLab the best, most complete DevSecOps platform on the planet. While we continue adding features to the product we must also work to identify technical debt and bring it to the prioritization discussion. We expect that Engineering managers are already addressing technical debt that is group specific with their Product Manager.
This coordination and prioritization requires a lot of work and effort to provide the right data and make the right decisions. We will focus on a variety of factors, but top of mind will be our parent department's direction to be customer focused.
We will follow our parent department Engineering lead.
The development team is responsible for developing products in the following categories:
The following people are permanent members of the Development Department:
The following members of other functional teams are our stable counterparts:
| Person | Role |
|---|---|
| Kalyani Yerraguntla | Learning & Development Program Manager |
This is the breakdown of our department by section and by stage.
This is the stack-up of our engineers, by level.
Aligned with the company-wide promotion cadence, Development utilizes a quarterly process to collect, validate, approve, review all promotion proposals prior to them being added via the company-wide process. The goal of this quarterly promotion projection and review is to:
Development adheres to the company-wide quarterly timeline outlined here as our SSOT.
The Development Department has an additional formal step built in to our promotion process beyond what the company is currently adhering to through our peer review process. Ahead of the commencement of the Calibration stage of our process, all promotion documents should be peer reviewed by a Senior Manager or Director. The due date to complete the peer review is before the scheduled Calibration session.
FY'23 Calibration sessions:
Calibration session attendees are the following team members: Senior Managers, Directors, Sr. Directors, VP, and Development's aligned People Business Partner. Leaders are welcome to conduct Calibration sessions prior to the scheduled sessions above with their sub-departments as well (though this is not a requirement).
In addition to the company-wide calibration preparation, for the Development department we also ask that leaders come prepared to discuss:
We use a peer review process to collaborate on proposed promotions.
Company-wide guidelines on the Talent Assessment can be found here. The company timeline for the process remains SSOT, the guidelines are below are meant to:
We will be reviewing outliers for Performance/Growth Potential and anyone identified as Key Talent for calibration this cycle. Formal calibration will take place at the Senior Manager, Director, and VP levels. The thought process around who qualifies as an "outlier" for Performance and Growth Potential is outlined here.
Calibration and assessment are two different steps in the process. The assessment phase is the process of assessing each team member to determine their Performance and Growth Potential, whereas the calibration phase (occurring after initial assessments are made) is when management calibrates across the stage/org/level/etc. to discuss and align on assessments. Every team member should be assessed and have supporting points to justify those assessments - but to make calibration sessions more focused and scalable, we focus on outliers.
For Development specifically, we will calibrate:
Exceeding for PerformancePerforming or Exceeding for Performance aligned with eligibility guidance or Growing or Exceeding for Growth PotentialIt's important to note that while these will be our focus areas for calibration sessions, managers should feel free to raise any team member's assessment up for discussions if they have any questions or concerns. Calibrating outliers is not a limitation, but rather a structural adjustment to ensure this process is scalable and focused.
Note: If individual teams want to calibrate every individual, they have the ability to do this/organize/structure separately, but the due dates remain in place across the department to ensure we have enough time to review and calibrate at the various levels in the company.
For team members who have assumed an Acting or an Interim role, we will assess team members aligned with their permanent positions (I.E. not the Acting or Interim position).
As the Talent Assessment impacts compensation, and Acting/Interim periods are not permanent, in the instance that a team member does not end up moving into the Acting/Interim role permanently, we would not want to have their compensation impacted by a temporary position.
In addition to the calibration session pre work on the Talent Assessment page, we ask that you complete the following:
Directors should conduct sub-department level calibration sessions ahead of the department-wide Development calibration sessions. In addition to calibrating on initial assessments, Directors are responsible for ensuring their respective teams assessments fall within the expected distribution ahead of the department-wide calibration session for efficiency.
Performance and Growth Potential notes need to be added to the session agenda doc for each team member at least 3 business days before the synchronous calibration session.
Managers should include 2-3 supporting points for each team member for the assessment under each section in the agenda doc notes to help support the “why” behind the assessment. Note: The overviews in the agenda document for calibration are meant to provide enough of an overview so peers have an understanding of the "why", while simultaneously not overwhelming with information and decreasing efficiency of the session. Please limit to 2-3 supporting points per team member.
Key Talent assessment need to be completed and added to the session agenda doc at least 3 business days before the live calibration session.
While calibrations and ultimate determinations will take place at Senior Manager+ level, inital recommendations begin in Workday and sit with team member's direct managers aligned with the process
We will calibrate by level within the Development department as a whole to ensure we have consistency and visibility across sub-departments.
| Session Number | Attendees | Calibration Level Focus | Session Date | Timezone Alignment | Duration |
|---|---|---|---|---|---|
| Session 1 | Senior Managers+ (people managers only) + PBP | Intermediate level outliers | 2021-11-07 | APAC/Americas | 1.5 hours |
| Session 2 | Senior Managers+ (people managers only) + PBP | Senior level outliers | 2021-11-08 | EMEA/Americas | 1.5 hours |
| Session 3 | Senior Managers+ (people managers only) + PBP | Senior level outliers | 2021-11-08 | APAC/Americas | 1.5 hours |
| Session 4 | Senior Managers+ (people managers only) + PBP | Senior level outliers | 2021-11-09 | EMEA/Americas | 1.5 hours |
| Session 5 | Senior Managers+ (people managers only) + PBP | Staff/EM/Principal/Distinguished level outliers | 2021-11-09 | APAC/Americas | 1.5 hours |
| Session 6 | Senior Managers+ (people managers only) + PBP | Intermediate level outliers | 2021-11-10 | EMEA/Americas | 1.5 hours |
| Session 7 | Senior Managers+ (people managers only) + PBP | Staff/EM/Principal/Distinguished level outliers | 2021-11-15 | EMEA/Americas | 1.5 hours |
| Session 8 | Senior Managers+ (people managers only) + PBP | Staff/EM/Principal/Distinguished level outliers | 2021-11-15 | APAC/Americas | 1.5 hours |
| Session 9 | Director+ (people managers only) + PBP | Calibrate Senior Manager level outliers | 2021-11-16 | EMEA/Americas | 2 hours |
| Session 10 | VP and PBP | Calibrate Director/Senior Director and Engineering Fellow level outliers | 2021-11-17 | Americas | 2 hours |
The company-wide Talent Assessment timeline can be found here and is SSOT. Below are level-specific calibration due dates embedded within the overall timeline for the Development Department.
The SSOT timeline for the upcoming Annual Compensation Review can be found here. Below you will find additional dates specific to the Development department to ensure all levels have time to review as we move through the process.
Phase 1 (cash only):
Phase 2 (equity only):
In this handbook page we document the process that the development department follows, including planning budget, candidate sourcing, interview process, contracting and onboarding.
Welcome to GitLab! We are excited for you to join us. Here are some curated resources to get you started:
The quad members (UX/Design, Quality, Product Management, Development) utilizing this process should focus on:
As long as the quad achieves these goals, they are encouraged to apply the process as appropriate based any unique characteristics of their group and also tailor the process based on how the team prefers to operate.
To support GitLab's long-term product health and stability while keeping the pace of new features for users, teams are asked to plan their milestones with an appropriate ratio of type::feature, type::maintenance, and type:bug work. When labeling if the label selection for an issue or merge request isn't obvious, don't spend more than 60 seconds to decide and make a best effort to choose the most appropriate label.
If one of these labels clearly doesn't apply for an issue, consider using the type::ignore label. This will exclude the issue from automation and dashboards used to do cross-functional prioritization and metrics tracking for the product. It is highly important we have accurate data, so please only use this label if the issue clearly does not pertain directly to Engineering changes to the product itself. This label will typically apply to issues used for planning or to track a process. For example, you could use the type::ignore label for a milestone planning issue where the issue's purpose is organization and will not have MRs directly associated with it.
A team's ratio might change over time and different teams may have different ratios. Factors that influence what ratio is appropriate for a given team include the product category maturity, the area of the product they are working in, and the evolving needs of GitLab the business. Teams should review labeling for accuracy and minimize the number of type::undefined items. This allows us to review the plans at the group, section, and company level with team members to ensure we appropriately prioritize based on cross-functional perspectives.
For more details on these three work types, please see the section on work type classification. The development EM is the DRI to ensure that the merge requests are accurateliy labeled.
Our backlog should be prioritized on an ongoing basis. Prioritization will be done via quad planning (collaboration between product, development, quality, UX) with PM as the DRI for the milestone plan. PMs, EMs, Quality, and UX will provide the following:
type::feature issuestype::maintenance issuestype::bug issues using the bug prioritization dashboardNote: UX-related work items would be prioritized in accordance with the appropriate sub-types. UX related bugs are included in the automated process (S1/2 and so on), UX-related maintenance items will be included in the EM's prioritized list, Product (feature) UX items will have been included as part of our normal Product Development Flow.
The DRIs of these three core areas will work collaboratively to ensure the overall prioritization of the backlog is in alignment with section direction or any other necessary product and business needs. If a team is not assigned a Product Designer then there is no UX counterpart needed for prioritization purposes. PMs will prioritize the final plan for a given milestone.
The Product Manager is responsible for planning each milestone. Product Managers are also responsible for ensuring that their team's target ratios are maintained over time.
Add the milestone (example) to review the milestone plan. The board will show the number of issues and cumulative issue weights for type::feature, type::maintenance, and type::bug issues.
Cross-functional reviews will be done at the group, stage/section, and company level.
When the data is up-to-date and accurate. See the timeline
Review the dashboard filtered for the review scope (group, section, etc).
Context:
Maintenance/quality:
Features:
Trends:
The review collaboration can be done in a way that's most effective for the group, either synchronously (e.g. scheduled recurring call) or asynchronously (e.g. such as in retrospective issues).
Required collaborators from the quad for the group are:
The managers of the required collaborators should be included as optional participants.
The stage/section review is coordinated by each direct report of the VP of Development.
Required collaborators from the quad for the stage/section are:
Optional collaborators who should be invited but not required to participate:
The collaboration should be async first but include an optional sync review amongst stakeholders.
The name of the meeting and associated agenda document should be clearly defined so that the invitees can decide if they should attend.
The company-wide review is coordinated by the VP of Development.
Required collaborators from the quad for the stage/section are:
Optional collaborators who should be invited but not required to participate:
Issues that impact code in another team's product stage should be approached collaboratively with the relevant Product, UX, and Engineering managers prior to work commencing, and reviewed by the engineers responsible for that stage.
We do this to ensure that the team responsible for that area of the code base is aware of the impact of any changes being made and can influence architecture, maintainability, and approach in a way that meets their stage's roadmap.
At times when cross-functional, or cross-departmental architectural collaboration is needed, the GitLab Architecture Evolution Workflow should be followed.
At GitLab we value freedom and responsibility over rigidity. However, there are some technical decisions that will require approval before moving forward. Those scenarios are outlined in our required approvals section.
Development's headcount planning follows the Engineering headcount planning and long term profitability targets. Development headcount is a percentage of overall engineering headcount. For FY20, the headcount size is 271 or ~58% of overall engineering headcount.
We follow normal span of control both for our managers and directors of 4 to 10. Our sub-departments and teams match as closely as we can to the Product Hierarchy to best map 1:1 to Product Managers.
While we try to work as much as possible async, the Development department leadership does meet synchronously on a cadence of weekly. This meeting coordinates initiatives, communicates relevant information, discusses more difficult decisions, and provides feedback on how we are progressing as an organization. As part of this meeting, we discuss our culture of reliability monthly. This was part of the agenda spawned from an initiative we took up in August of 2021. We want to make sure we keep the organization healthy when thinking about reliability in every part of our work.
This section applies to those who report to the VP of Development
The following is a non exhaustive list of daily duties for engineering directors, while some items are only applicable at certain time, though.
The schedule below covers the period from 2022-12-19 to 2023-01-02. To reach the emergency contacts, please send Slack as well as Text(when possible) messages for timely responses. The phone numbers can be found in Slack profiles.
| Name | Time Zone | Available Dates |
|---|---|---|
| Chun Du | GMT-8 (Portland, OR) | 2022-12-19 ~ 2023-01-02 |
| Sam Goldstein | GMT-8 (Portland, OR) | December 19 - 26 (note you can also notify me via Pagerduty |
| Wayne Haber | GMT-5 (NYC) | Available for escalations via SMS number in Slack profile. Also, see coverage plan |
| Christopher Lefelhocz | GMT-6 (San Antonio, TX) | 2022-12-19 - 2022-12-22, 2022-12-27 - 2023-01-02 |
| Darva Satcher | GMT-5 | 2022-12-23 - 2023-01-06 |
| Phil Calder | GMT+13 (Wellington, New Zealand) | 2022-12-19 ~ 2022-12-21 |
| Tim Zallmann | GMT+1 (Vienna, Austria) | 2022-12-19 - 2022-12-22, 2022-12-28 - 2012-12-30 (for needed EMEA emergency coverage) |
| Jerome Ng | GMT-8 (Vancouver, Canada) | 2022-12-19 ~ 2023-01-02 |
| Michelle Gill | GMT-6 (Texas, USA) | 2022-12-19 - 2022-12-23 |
| Nick Nguyen | GMT-6 (Minnesota, USA) | 2022-12-19 - 2022-12-23, 2022-12-27 - 2022-01-02 |
In general, OKRs flow top-down and align to the company and upper level organization goals.
For managers and directors, please refer to a good walk-through example of OKR format for developing team OKRs. Consider stubbing out OKRs early in the last month of the current quarter, and get the OKRs in shape (e.g. fleshing out details and making them SMART) no later than the end of the current quarter.
It is recommended to assess progress weekly.
Below are tips for developing individual's OKRs:
Engineering Allocation require us to track goals with more diligence and thought. We need confidence that we’re making correct decisions and executing well to these initiatives. As such, you will see us reviewing these more closely than other initiatives. We will meet on a cadence to review these initiatives and request additional reporting to support the process. Possible requests for additional data:
We will hold Engineering Allocation Checkpoints on a cadence. The recommended cadence is weekly.
We track Engineering Allocation roadmaps. To use this effectively, roadmaps must have correct dates for their epic and weights assigned to issues. If a team does not normally use weights, then assign each issue a weight of 1 (all issues are equal).
Each team needs to demonstrate how their allocation is being used. This is done to verify we are not over/under investing for a given initiative. This can be done via assignment (people assigned to work) and/or issues assigned. We will track issues and MRs and see as a percentage how that compares to the overall teams work.
The GitLab application is built on top of many shared services and components, such as PostgreSQL database, Redis, Sidekiq, Prometheus and so on. These services are tightly woven into each feature's rails code base. Very often, there is need to identify the DRI when demand arises, be it feature request, incident escalation, technical debt, or bug fixes. Below is a guide to help people quickly locate the best parties who may assist on the subject matter.
There are a few available models to choose from so that the flexibility is maximized to streamline what works best for a specific shared service and component.
The shared services and components below are extracted from the GitLab product documentation.
| Service or Component | Sub-Component | Ownership Model | DRI (Centralized Only) | Ownership Group (Centralized Only) | Additional Notes |
|---|---|---|---|---|---|
| Alertmanager | Centralized with Specific Team | @twk3 | Distribution | Distribution team is responsible for packaging and upgrading versions. Functional issues can be directed to the vendor. | |
| Certmanager | Centralized with Specific Team | @twk3 | Distribution | Distribution team is responsible for packaging and upgrading versions. Functional issues can be directed to the vendor. | |
| Consul | |||||
| Container Registry | Centralized with Specific Team | @dcroft | Package | ||
| Email - Inbound | |||||
| Email - Outbound | |||||
| GitLab K8S Agent | Centralized with Specific Team | @nicholasklick | Configure | ||
| GitLab Pages | Centralized with Specific Team | David O'Regan @oregand | Editor | ||
| GitLab Rails | Decentralized | DRI for each controller is determined by the feature category specified in the class. app/controllers and ee/app/controllers | |||
| GitLab Shell | Centralized with Specific Team | @sean_carroll | Create:Source Code | Reference | |
| HAproxy | Centralized with Specific Team | @sloyd | Infrastructure | ||
| Jaeger | Centralized with Specific Team | @sloyd | Infrastructure:Observability | Observability team made the initial implementation/deployment. | |
| LFS | Centralized with Specific Team | @sean_carroll | Create:Source Code | ||
| Logrotate | Centralized with Specific Team | @twk3 | Distribution | Distribution team is responsible for packaging and upgrading versions. Functional issues can be directed to the vendor. | |
| Mattermost | Centralized with Specific Team | @twk3 | Distribution | Distribution team is responsible for packaging and upgrading versions. Functional issues can be directed to the vendor. | |
| MinIO | Decentralized | Some issues can be broken down into group-specific issues. Some issues may need more work identifying user or developer impact in order to find a DRI. | |||
| NGINX | Centralized with Specific Team | @twk3 | Distribution | ||
| Object Storage | Centralized with Specific Team | @lmcandrew | Scalability::Frameworks | ||
| Patroni | General except Geo secondary clusters | Centralized with Specific Team | @twk3 | Distribution | |
| Geo secondary standby clusters | Centralized with Specific Team | @nhxnguyen | Geo | ||
| PgBouncer | Centralized with Specific Team | @twk3 | Distribution | ||
| PostgreSQL | PostgreSQL Framework and Tooling | Centralized with Specific Team | @alexives | Database | Specific to the development portion of PostgreSQL, such as the fundamental architecture, testing utilities, and other productivity tooling |
| GitLab Product Features | Decentralized | Examples like feature specific schema changes and/or performance tuning, etc. | |||
| Prometheus | Decentralized | Each group maintains their own metrics. | |||
| Puma | Centralized with Specific Team | @pjphillips | Application Performance | ||
| Redis | Decentralized | DRI is similar to Sidekiq which is determined by the feature category specified in the class. app/workers and ee/app/workers | |||
| Sentry | Decentralized | DRI is similar to GitLab Rails which is determined by the feature category specified in the class. app/controllers and ee/app/controllers | |||
| Sidekiq | Decentralized | DRI for each worker is determined by the feature category specified in the class. app/workers and ee/app/workers | |||
| Workhorse | Centralized with Specific Team | @sean_carroll | Create:Source Code |
For a list of resources and information on our GitLab Learn channel for Development, consult this page.
In late June 2019, we moved from a monthly release cadence to a more continuous delivery model. This has led to us changing from issues being concentrated during the deployment to a more constant flow. With the adoption of continuous delivery, there is an organizational mismatch in cadence between changes that are regularly introduced in the environment and the monthly development cadence.
To reduce this, infrastructure and quality will engage development via SaaS Infrastructure Weekly and Performance refinement which represent critical issues to be addressed in development from infrastructure and quality.
Refinement will happen on a weekly basis and involve a member of infrastructure, quality, product management, and development.
Execution of a Global prioritization can take many forms. This is worked with both Product and Engineering Leadership engaged. Either party can activate a proposal in this area. The options available and when to use them are the following:
When Development collaborates with Support it provides invaluable insight into how customers are using the product and the challenges they run into. A few tips to make the process efficient:
Team members in some job families contribute to incident management directly through an on-call schedule for Incident Managers. Team members should complete onboarding so they can be added to the schedule when needed. These frequently asked questions cover exemptions and changing shifts.
Because our teams are working in separate groups within a single application, there is a high potential for our changes to impact other groups or the application as a whole. We have to be cautious not to inadvertently impact overall system quality but also availability, reliability, performance, and security.
An example would be a change to user authentication or login, which might impact seemingly unrelated services, such as project management or viewing an issue.
Far-reaching work is work that has wide-ranging, diffuse implications, and includes changes to areas which will:
If your group, product area, feature, or merge request fits within one of the descriptions above, you must seek to understand your impact and how to reduce it. When releasing far-reaching work, use a rollout plan. You might additionally need to consider creating a one-off process for those types of changes, such as:
Some areas have already been identified that meet the definition above, and may consider altered approaches in their work:
| Area | Reason | Special workflows (if any) |
|---|---|---|
| Database migrations, tooling, complex queries, metrics | impact to entire application The database is a critical component where any severe degradation or outage leads to an S1 incident. |
Documentation |
| Sidekiq changes (adding or removing workers, renaming queues, changing arguments, changing profile of work required) | impact to multiple services Sidekiq shards run groups of workers based on their profile of work, eg memory-bound. If a worker fails poorly, it has the potential to halt all work on that shard. |
Documentation |
| Redis changes | impact to multiple services Redis instances are responsible for sets of data that are not grouped by feature category. If one set of data is misconfigured, that Redis instance may fail. |
|
| Package product areas | high percentage of traffic share | |
| Gitaly product areas | high percentage of traffic share | |
| Create: Source Code product areas | high percentage of traffic share. Special attention should be paid to Protected Branches, CODEOWNERS, MR Approvals, Git LFS, Workhorse and the git over SSH / gitlab-sshd interfaces. Please contact the EM (@sean_carroll) or PM (@tlinz) if you are unsure. | |
| Pipeline Execution product areas | high percentage of traffic share | Documentation |
| Authentication and Authorization product areas | touch multiple areas of the application | Documentation |
| Compliance product areas | potentially have legal, security, or compliance consequences | Code Review Documentation |
| Workspace product areas | touch multiple areas of the application | Documentation |
| Specific fulfillment product areas | potentially impact revenue | |
| Runtime language updates | impacts to multiple services | Ruby Upgrade Guidelines |
| Application framework updates | impacts to multiple services | Rails Upgrade Guidelines |
ClickHouse usage by Monitor:Observability group
If development is the DRI or actively participating in a Customer Account Escalation, consider the following:
Note: books in this section can be expensed.
Interested in reading this as part of a group? We occasionally self-organize book clubs around these books and those listed on our Leadership page.