GitLab has a Three-Year Strategy, and we're excited to see every member of the Engineering division contribute to achieving it. Whether you're creating something new or improving something that already exists, we want you to feel empowered to bring your best ideas for influencing the product direction through improved scalability, usability, resilience, and system architectures. And when you feel like you need to expand your knowledge in a particular area, know that you're supported in having the resources to learn and improve your skills.
Having a product that people enjoy using is critical to our company’s success. Many users rely on GitLab every single workday to create their own great products. When we offer experiences that make their jobs faster and easier, we support them in building success in their own careers—a responsibility that we take seriously. And when our users are successful, we’re more successful, too, as evidenced by a McKinsey study showing 32% higher revenue growth in a 5-year period for companies that focus on great design.
That’s why part of our FY23 focus is to raise our System Usability Scale score through efforts like burning down significant (Sev 1 and Sev 2) usability issues, addressing Usability Benchmark insights, and migrating our UI to single-source-of-truth Pajamas components for better consistency, accessibility, and general usability.
We have a 3-year goal of reaching 1,000 monthly contributors as a way to mature new stages, add customer-desired features that aren’t on our roadmap, and even translate our product into multiple languages. To build towards that goal, in FY23, we’re enhancing our Contributor Education to help support high-quality contributions that align with our approach to product development. We’re also improving our response time for contributions and providing timely feedback for a better contributor experience. Lastly, we’re focused on a step-change increase of contributors by creating a Leading Organizations program tailored for full-time contributors.
Diverse teams perform better. They provide a sense of belonging that leads to higher levels of trust, better decision making, and a larger talent pool. They also focus more on facts, process facts more carefully, and are more innovative. By hiring globally and increasing the numbers of women and ethnic minorities in the Engineering division, we’re helping everyone bring their best selves to work.
Hiring is still a top priority in FY23, and we're excited to continue hiring people who are passionate about our product and have the skills to make it the best DevSecOps tool in the market. Our current focus areas include reducing the amount of time between offer and start dates and hiring a diverse team (see above). We're also implementing industry-standard approaches like structured, behavioral, and situational interviewing to help ensure a consistent interview process that helps to identify the best candidate for every role. We’re excited to have a recruiting org to partner with as we balance the time that managers spend recruiting against the time they spend investing in their current team members.
As expected, a large part of our focus in FY23 is on improving our product. Following are some large projects to note.
For Enterprise customers, we’re refining our product to meet the levels of security and reliability that customers rightfully demand from SaaS platforms (SaaS Reliability). We’re also providing more robust utilization metrics to help them discover features relevant to their own DevOps transformations (Usage Reporting) and offering the ability to purchase and manage licenses without spending time contacting Sales or Support (E-Commerce and Cloud Licensing). Lastly, in response to Enterprise customer requests, we're adding features to support Suggested Reviewers, better portfolio management through Work Items, and Audit Events that provide additional visibility into user passive actions.
For Free Users, we’re becoming more efficient with our open core offering, so that we can continue to support and give back to students, startups, educational institutions, open source projects, GitLab contributors, and nonprofits (SaaS Free User Efficiency).
For Federal Agencies, we’re obtaining FedRAMP certification to strengthen confidence in the security standards required on our SaaS offering. This is a mandated prerequisite for United States federal agencies to use our product.
For Hosted Customers, we're supporting feature parity between Self-Managed and GitLab Hosted environments through the Workspace initiative. We're also launching GitLab Dedicated for customers who want the flexibility of cloud with the security and performance of a single-tenant environment.
For customers using CI/CD, we're expanding the available types of Runners to include macOS, Linux/Docker, and Windows, and we're autoscaling build agents.
In alphabetical order:
GitLab Engineering values clear, concise, transparent, asynchronous, and frequent communication. Here are our most important modes of communication:
As part of a fully-distributed organization such as GitLab, it is important to stay informed about engineering-led initiatives. We employ multimodal communication, which describes the minimum set of communication channels we'll broadcast to.
For the Engineering division, any important initiative will be announced in:
#engineering-fyi
Other Slack channels that are Engineering focused and are good sources of information:
#cto#development#incubation-eng#production#quality#incident-management#support_gitlab-com#support_self-managedThere is no requirement to join all of these channels. It is up to the person sharing to ensure that the same message is shared across all channels. Ideally, this message should be a one sentence summary with a link to an issue to allow for a single source of truth for any feedback.
The #engineering-fyi is used for large scale announcements and to drive views of the engineering week in review document. Everyone can contribute an announcement as long as the following criteria are met:
The posting model here is one of trusting judgement of the individual making the announcement. You do not need to ask for permission to post.
There are seven departments within the Engineering Division:
See the Cross-Functional Prioritization page for more information.
To maintain high availability, Engineering runs a weekly SaaS Availailbity standup to:
Each week the Infrastructure team reports on incidents and key metrics. Updating these items at the top of the Engineering Allocation Meeting Agenda is the responsibility of the Engineering Manager for the General Squad in Reliability.
For development section, updates on current status of:
Groups under Feature Change Locks should update progress synchronously or asynchronously in the weekly agenda. The intention of this meeting is to communicate progress and to evaluate and prioritise escalations from infrastructure.
Feature Change Locks progress reports should appear in the following format in the weekly agenda:
FCL xxxx - [team name]
<issue link><issue link><issue link><link to Timeline tab of the Incident issue>
We will enact a localized feature change lock (FCL) anytime there is an S1 or public-facing (status page) S2 incident on GitLab.com (including the License App, CustomersDot, and Versions) determined to be caused by an engineering department change. The team involved should be determined by the author, their line manager, and that manager's other direct reports.
If the incident meets the above criteria, then the manager of the team is responsible for the planning and execution of their team's FCL and informing their manager (e.g. Senior Manager / Director) that the team will focus efforts towards an FCL. The manager will also provide updates at the SaaS Availability Weekly Standup.
If the team believes there does not need to be an FCL, approval must be obtained from either the VP of Infrastructure or VP of Development.
Direct reports involved in an active borrow should be included if they were involved in the authorship or review of the change.
The purpose is to foster a sense of ownership and accountability amongst our teams, but this should not challenge our no-blame culture.
Rough guidance on timeline is provided here to set expectations and urgency for an FCL. We want to balance moving urgently with doing thoughtful important work to improve reliaiblity. Note that as times shift we can adjust accordingly. The DRI of an FCL should pull in the timeline where possible.
The following bulleted list provides a suggested timeline starting from incident to completion of the FCL. "Business day x" in this case refers to the x business day after the incident.
During the FCL, the team(s) exclusive focus is around reliability work, and any feature type of work in-flight has to be paused or re-assigned. Maintainer duties can still be done during this period and should keep other teams moving forward. Explicitly higher priority work such as security and data loss prevention should continue as well. The team(s) must:
#fcl-incident-[number], with members
[Group Name] FCL for Incident ####closing ceremony upon completing the FCL to review the retrospectives and celebrate the learnings.
After the Incident Review is completed, the team(s) focus is on preventing similar problems from recurring and improving detection. This should include, but is not limited to:
Examples of this work include, but are not limited to:
Any work for the specific team kicked off during this period must be completed, even if it takes longer than the duration of the FCL. Any work directly related to the incident should be kicked off and completed even if the FCL is over. Work paused due to the FCL should be the priority to resume after the FCL is over. Items created for other teams or on a global level don't affect the end of the FCL.
A stable counterpart from Infrastructure will be available to review and consult on the work plan for Development Department FCLs. Infrastructure FCLs will be evaluated by an Infrastructure Director.
Team members are welcome to run Folding@home on their company provided computers. Folding@home is a distributed computing network that is searching for therapies for the COVID-19 respiratory illness among other diseases. We recommend running it at night if you have high daily compute workloads. Also keep your computer plugged in. We considered potential security and hardware implications in this issue.
If you would like to join a team with other GitLab team members, there is a GitLab Team Members team for Folding@home. When setting up or changing your Folding@home identity, you can add team 245256. This is not a competition, but simply to track how much our team members have contributed overall. You can view our statistics on our team page. You can discuss with other GitLab team members in the #folding-at-home slack channel.
Please reference our internal hiring repository for internal best practices and guidelines.
Budgeted new headcount review occurs on a regular basis across Engineering Leadership, Talent Acquisition, Finance Business Partners (FPB), and People Business Partners (PBP). The process to open a new headcount is Talent Acquisition and Finance-driven, with a focus on ensuring communication, consistency and collaboration between the Engineering and People teams.
Talent Acquisition: Talent Acquisition will use the New Headcount Issue template to request a GHPID via the R&D Headcount Project in GitLab. Details to be included: details of the role, Hiring Manager, target start date, and budget initiative (if applicable). The Finance Business Partner, Department VP, PBP, Recruiting Manager and Recruiter will be tagged. The following labels will be applied: Department, “New Headcount” and “Workflow: Finance GHPID.”
Department VP: Department VP will review the Issue, provide context or guidance as necessary and approve the role as recruiting ready for GHPID creation.
Finance Business Partner: The FBP will create a GHPID for the forecasted role in Adaptive, add the GHPID to the Issue and mark the step as complete signaling the forecast has been updated. If the FBP needs more information they will ask the department leader before providing a GHPID. If there is an increase in the expense for the new headcount request from current forecast (this could be driven by an earlier start date, increase in level, or change in roles), they will work with the department leader at that time to identify tradeoff options to fund the increase before providing the GHPID and approving. Once the GHPID is approved the FBP will apply the “Workflow: Recruiting” label to the issue.
Recruiter: The Recruiter will create the Greenhouse requisition and report the new requisition number on the Issue, checking off when complete, and adding the “Workflow: Req Now Hiring” label to the Issue. The Recruiter will communicate the role Kickoff process with the Hiring Manager, send a Kickoff issue for completion and advertise the role internally.
Hiring Manager: The Hiring Manager is notified of the role approval by receiving the Kickoff Issue from the Recruiter. The kickoff issue must be completed within 5 days, with a synchronous kickoff meeting scheduled to go through the kickoff and take a formal job brief.
This Backfill process is to be used when a team member departs from GitLab or they transfer to a different role. This process ensures alignment between the Department Head, Finance Business Partner and Talent Acquisition. For departures, a backfill can only be opened once a departure or resignation is official where we've received written confirmation of the departure including the last working day and the offboarding process has been completed in Workday. For transfers, a backfill can only be opened once a transfer is official where an offer letter stating the transfer date has been completed.
Manager: Manager is made aware of team member departure (either through our voluntary, involuntary offboarding) or transfer process.
Manager: Manager shares the offboarding or transfer with their People Business Partner and direct manager, up to their Department Head. The manager, team member, and/or Team Member Relations (where appropriate) should follow the relevant offboarding steps in Workday dependent on the situation. PBP and Department leadership can be notified via a private Slack channel if needed. Finance Business Partners (FBP) do not have to be included in these private channels.
Talent Acquisition: TA Leadership will have access to a Workday Attrition Notification Report that will be reviewed 2x weekly to identify backfill role needs. TA Leadership will allocate the backfill role to the appropriate Recruiter. Within 30 days of the Workday termination date the Recruiter will use the Utilizing Backfill Headcount Issue template to request a GHPID via the R&D Headcount Project in GitLab. The Finance Business Partner, eGroup Leader, Department VP, People Business Partner, and Recruiting Manager will be tagged in this issue, and the Department and “backfill headcount” labels will be added to the Issue.
eGroup Leader: The eGroup leader will review the Issue and determine how this backfill headcount will be utilized (backfill in kind, reallocate to a role within the department, reallocate to a different department etc.). The eGroup leader will provide context or guidance as necessary and approve the role as recruiting ready for GHPID creation. After approval, the TA Leader will add the “Workflow: Finance GHPID Allocation” label.
Finance Business Partner: The “Workflow: Finance GHPID Allocation” label indicates that the role is ready for FBP (Finance Business Partner) review and GHPID allocation. FBP will create the backfill headcount in Adaptive, and generate a GHPID for the backfill headcount. They will then add the GHPID to the Issue and mark the step as complete. This signals that the forecast has been updated with the backfill. If the FBP needs more information they will ask the TA leadership or eGroup or Departmental leader at this time before providing a GHPID. If there is a change in the associated backfill expense, they will work with the leader to identify tradeoff options before approving.
Recruiter: The TA Leader will tag the appropriate Recruiter who will create the Greenhouse requisition and report the new requisition number on the Issue, checking off when complete. The Recruiter will add the “Workflow: Req Now Hiring” label to the Issue and ensure the Hiring Manager has been notified to begin the kickoff process.
Department Heads and PBPs: If two Department Heads agree to move a new headcount between their teams each leader should reach out to their respective PBPs to discuss org structure and/or org planning considerations. PBPs will share relevant information (org design/level/placement) with Finance and Talent Acquisition through the Triad process.
Talent Acquisition: TA will own the process of creating an Approved Needing Swap issue in the Headcount Project to confirm that both departments are in agreement with the move.
Finance Business Partner: The FBP will create a GHPID for the forecasted role in Adaptive, add the GHPID to the Issue and mark the step as complete, signaling the forecast has been updated. If more information is required, the FPB will reach out to the department leader or PBP before providing a GHPID.
People Business Partner: After a GHPID has been allocated, the PBP will ensure that both the "Cost Center" and “Supervisory Org” dimensions in Workday are updated and accurate.
The VP of Engineering and their direct reports track our highest priorities in the Engineering Management Issue Board, rather than to do lists, Google Doc action items, or other places. The reasons for this are:
Here are the mechanics of making this work:
Engineering Management label to get it on the board, and the department label to get it in progress (e.g. Development Department)CEO Interest label, please post it to #ceoThe Quality Department is the DRI for Engineering Performance Indicators. Work regarding KPI / RPI is tracked on the engineering metrics board and task process.
In GitLab Engineering we are serious about concepts like servant leadership, over-communication, and furthering our company value of transparency. You may have joined GitLab from another organization that did not share the same values or techniques. Perhaps you're accustomed to more corporate politics? You may need to go through a period of "unlearning" to be able to take advantage of our results-focused, people-friendly environment. It takes time to develop trust in a new culture.
Less common, but even more important, is to make certain you don't unintentionally bring any mal-adaptive behaviors to GitLab from these other environments.
We encourage you to read the engineering section of the handbook as part of your onboarding, ask questions of your peers and managers, and reflect on how you can help us better live our culture:
We manually verify that our code works as expected. Automated test coverage is essential, but manual verification provides a higher level of confidence that features behave as intended and bugs are fixed.
We manually verify issues when they are in the workflow::verification state.
Generally, after you have manually verified something, you can close the associated issue.
See the Product Development Flow to learn more about this issue state.
We manually verify in the staging environment whenever possible. In certain cases we may need to manually verify in the production environment.
If you need to test features that are built for GitLab Ultimate then you can get added to the issue-reproduce group on production and staging environments by asking in the #development Slack channel. These groups are are on an Ultimate plan.
Before the beginning of each fiscal year, and at various check points throughout the year, we plan the size and shape of the Engineering and Product Management functions together to maintain symmetry.
The process should take place in a single artifact (usually a spreadsheet, current spreadsheet), and follow these steps:
Note: Support is part of the engineering function but is budgeted as 'cost of sales' instead of research and development. Headcount planning is done separately according to a different model.
The non support related departments within Engineering (Development, Infrastructure, Quality, and Security) have an expense target of 20% as a percentage of revenue.
The Support target is 10% as a percentage of revenue.
The PlatoHQ Program has a total of 10 Engineering Managers/Senior IC's participating. The program exists of both self-learning via an online portal and 1-1 sessions with a mentor.
The 7CTOs Program is run with 4 Senior leaders in Engineering. The program exists of peer mentoring sessions (forums) and effective network building.
The CTO is an executive sponsor for selected customers.
A shadow program is available to everyone in engineering (especially senior leaders) in order to have an opportunity to observe and participate in one of the executive sponsor meetings. Doing so can be a great way to hear directly from customers about what they like about GitLab and about what we can improve. (This program is similar in some ways to the CEO Shadow Program.
If you choose to be a shadow, your responsibilities will be:
To request to be a shadow: Post a message in the #cto Slack channel, indicate your timezone, and CC the CTO's EBA Kristie Thomas.
There is a program to find a mentor or to become a mentor at GitLab described on this handbook page.
You can find more information on this experimental program on the Development Director Shadow Program handbook page.
Engineers can gain significant insight from observing customers while they use the portions of the product that those engineers develop. Engineers interested in doing this should contact their team's assigned UX Researcher to get involved with upcoming customer sessions and access previously recorded sessions.
If a team does not have a UX Researcher assigned, they should reach out on the #research Slack channel to express interest. A UX Researcher can jump in and set them up to observe a participant session.
If there is a specific request to shadow a customer as they use the product, this should be coordinated through the team's assigned Product Manager.
We follow the below process when existing critical customer escalations requires immediate scheduling of bug fixes or development effort.
~"priority::1" regardless of severity~"critical-customer-escalation" is applied to the issueThe DRI can use the customer critical merge requests process to expedite code review & merge.
In most cases, a single engineer and maintainer review are adequate to handle a priority::1/severity::1 issue. However, some issues are highly difficult or complicated. Engineers should treat these issues with a high sense of urgency. For a complicated priority::1/severity::1 issue, multiple engineers should be assigned based on the level of complexity. The issue description should include the team member and their responsibilities.
| Team Member | Responsibility |
|---|---|
Team Member 1 |
Reproduce the Problem |
Team Member 2 |
Audit Code Base for other places where this may occur |
If we have cases where three or five or X people are needed, Engineering Managers should feel the freedom to execute on a plan quickly.
Following this procedure will:
Engineering is the primary advocate for the performance, availability, and security of the GitLab project. Product Management prioritizes all initiatives, so everyone in the engineering function should participate in the Product Management prioritization process to ensure that our project stays ahead in these areas. The following list should provide some guidelines around the initiatives that each engineering team should advocate for during their release planning:
Support Team Contributions label. You can filter on open MRs.GitLab makes use of a 'Canary' stage. Production Canary is a series of servers running GitLab code in a production environment. The Canary stage contains code functional elements like web, container registry and git servers while sharing data elements such as sidekiq, database, and file storage with production. This allows UX code and most application logic code to be consumed by a smaller subset of users under real world scenarios before being made available to all users on GitLab.com.
Information on canary testing has been moved to dedicated page covering the canary stage and how to use it
There are primarily two Slack channels which developers may be called upon to assist the production team when something appears to be amiss with GitLab.com:
#backend: For backend-related issues (e.g. error 500s, high database load, etc.)#frontend: For frontend-related issues (e.g. JavaScript errors, buttons not working, etc.)Treat questions or requests from production team for immediate urgency with high priority.
There are some engineering handbook topics that we cannot be publicly transparent about. These topics can be viewed by GitLab team members in the engineering section of the private handbook.
If you experience a page not found (404) error when attempting to access the internal handbook, you may need to register to use it via first browsing to the internal handbook authorization page.