Gitlab hero border pattern left svg Gitlab hero border pattern right svg

Engineering

Communication

GitLab Engineering values clear, concise, transparent, asynchronous, and frequent communication. Here are our most important modes of communication:

Keeping yourself informed

As part of a fully distributed organization such as GitLab, it is important to stay informed about engineering led initiatives. We employ multimodal communication, which describes the minimum set of communication channels we'll broadcast to.

For the Engineering department, any important initiative will be announced in:

If you frequently check any of these channels, you can consider yourself informed. It is up to the person sharing to ensure that the same message is shared across all channels. Ideally, this message should be a one sentence summary with a link to an issue to allow for a single source of truth for any feedback.

Prioritizing technical decisions

Please see the Product Management section that governs how they prioritize work, and also should guide our technical decision making.

  1. Security fixes
  2. Data-loss prevention
  3. Availability
  4. Fixing regressions (things that worked before)
  5. Velocity of new features, user experience improvements, technical debt, community contributions, and all other improvements
  6. Lastly, behaviors that yield higher predictability (because this inevitably slows us down)

Despite the high priority of velocity to our project and our company, there is one set of things we must prioritize over it: GitLab availability & security. Neither we, nor our customers, can run an Enterprise-grade service if we are willing to risk users productivity and data.

Our hundreds of Engineers collectively make thousands of independent decisions each day that can impact GitLab.com and our users and customers there. They all need to keep availability and security in mind as we endeavor to be the most productive engineering organization in the world. We can only move as fast as GitLab.com is available and secured. Availability of self-managed GitLab instances is also extremely important to our success, and this needs to happen in partnership with our customers' admins (whereas we are the admins for GitLab.com)

For security, we prioritize it more highly by having strict SLAs around priorities labels with security issues. This shows a security first mindset as these issues take precedence in a given timeframe.

The Importance of Velocity

Incremental Velocity and Measurement

Our velocity should be incremental in nature. It's derived from our MVC, which encourages "delivering the smallest possible solution that offers value to our users". This could be a small new feature, but also includes code improvements, fixing bugs, etc.

To measure this, we count and define the target here: MRs per Development Engineer which is a goal for managers and not ICs. Historically, we have seen this as high as 14-19 MRs per Product Development Engineer per Month.

Ten MRs per month per Product Development Engineer translates to roughly an MR every 1 1/2 business days with time for overhead. To attain this, Product Development Engineers are encouraged to:

Velocity over predictability

We optimize for shipping a high volume of user/customer value with each release. We do want to ship multiple major features in every monthly release of GitLab. However, we do not strive for predictability over velocity. As such, we eschew heavyweight processes like detailed story point estimation by the whole team in favor of lightweight measurements of throughput like the number of merge requests that were included or rough estimates by single team members.

There is variance in how much time an issue will take versus what you estimated. This variance causes unpredictability. If you want close to 100% predictability you have to take two measures:

  1. Invest more time in estimation to reduce that variance. The time spent estimating things could otherwise be used to create features.
  2. Leave a reserve of time with unscheduled work so you can accommodate the variance. According to Parkinson's law the work expands so as to fill the time available for its completion. This means that we're not adhering to our iteration value and that for the next cycle our estimates for comparable features will be larger.

Both measures reduce the overall velocity of shipping features. The way to prevent this is to accept that we don't want perfect predictability. Just like with our OKRs, which are so ambitious that we expect to reach about 70% of the goal, this is also fine for shipping planned features.

Note: This does not mean we place zero value on predictability. We just optimize for velocity first.

Balance refactoring and velocity

When changing an outdated part of our code (e.g. HAML views, jQuery modules), use discretion on whether to refactor or not. For long term maintainability, we are very interested in migrating old code to the consistent and preferred approach (e.g. Vue, GraphQL), but we're also interested in continuously shipping features that our users will love.

Aim to implement new modules or features with the preferred approach, but changing preexisting non-conforming parts is a gray area.

If the weight of refactoring and other constraints (such as time) risk threatening the availability of a feature, then strongly consider refactoring at another time. On the other hand, if the code in question has hurt availability or poses a threat to it, then strongly consider prioritizing refactoring. This is a balancing act and if you're not sure where your change should go (or whether you should do some refactoring before hand), reach out to another Engineer or Maintainer.

If it makes sense to refactor before implementing a new feature or a change, then please:

If it is decided not to refactor at this moment, then please:

Hiring Practices

We're currently in a time of rapid growth (Engineering grew 100% in 2018 and is planned to grow by 130% in 2019). But we value quality hires over meeting numbers in the hiring plan. We rely primarily on the judgement of our hiring managers to hold the bar high. But we also try to systematize as much as possible so our hiring practices are fair, transparent, and repeatable.

Shadowing VPE's interviews

Engineering Management Issue Board

The VP of Engineering and their direct reports track our highest priorities in the Engineering Management Issue Board, rather than to do lists, Google Doc action items, or other places. The reasons for this are:

Here are the mechanics of making this work:

Engineering OKR process

Here is the standard, company-wide process for OKRs. Engineering has some small deviations from (and extensions to) this process.

OKR Kickoff

This process should begin no later than two weeks before the end of the preceding quarter. And kickoff should happen on or before the first day of the new quarter.

  1. OKR owners should author new issues in the handbook project using the "Engineering OKR" description template
    • The issue title should be FY20-Q2 Organization Type OKR: Objective phrase => 0%
      • Type should be one of "IACV", "Product", or "Team"
      • e.g. FY20-Q3 Engineering Product OKR: Build our product vision => 0%
    • Update the issue description
      • Add your Key results phrases to the issue description. Valid Key results are:
        • Raising a KPI from one specific value to another
        • Building out a new KPI
        • Failing either of the first two… Completing a high-profile project with specific outcomes
        • e.g. * Raise first reply-time SLA for premium from 92% to 95% => 0%
      • Add your manager's and your direct report's handles to the CC line
    • Assign the issue to yourself
    • Set the due date to the last day in the quarter
    • Apply the appropriate labels to make sure it appears in your appropriate column of our management board
    • Interlink related OKRs (usually by OKR type) of your manager and direct reports using the related issues field
  2. Get approval prior to the first day of the quarter from your manager
    • For the VPE and their direct reports:
      • Do an MR to that quarter's markdown handbook page
        • * Department: [Objective phrase](https://placeholder.com/) => 0% e.g. * Support: Raise first reply-time SLA for premium from 92% to 95% => 0%
        • Indent department level OKRs underneath the Engineering Division OKRs
        • One line for each objective
      • Assign the Mr to the VPE and address changes asynchronously like a code review
      • Discuss in 1:1 if needed
    • For everyone else: Ask you manager to do an async review of your issues via Slack or email and address any changes. Alternatively, discuss in a 1:1.
  3. Communicate dependencies to other divisions, departments, or teams. Encourage them to take on corollary OKRs.

OKR Status

OKR Retrospection

This process should begin on the first day of the subsequent quarter, and complete no later that two weeks after.

  1. OKR owners should score their OKRs in the issue
    • Update the overall score in the issue title.
    • Update the individual key result scores in the issue description.
  2. OKR owners should retrospect in the issue description.
  3. OKR owners should do an MR to that quarter's OKR page with just the final scores after the objective phrase/link (e.g. => 70%) and assign it to their direct manager for review.
  4. The manager should review each individual issue, ask any questions, and merge
    • The OKR owner should incorporate any manager feedback like in a code review

Unlearning Previous Corporate Cultures

In GitLab Engineering we are serious about concepts like servant leadership, over-communication, and furthering our company value of transparency. You may have joined GitLab from another organization that did not share the same values or techniques. Perhaps you're accustomed to more corporate politics? You may need to go through a period of "unlearning" to be able to take advantage of our results-focused, people-friendly environment. It takes time to develop trust in a new culture.

Less common, but even more important, is to make certain you don't unintentionally bring any mal-adaptive behaviors to GitLab from these other environments.

We encourage you to read the engineering section of the handbook as part of your onboarding, ask questions of your peers and managers, and reflect on how you can help us better live our culture:

Dogfooding

We dogfood everything. Based on our product principles, it is the Engineering division's responsibility to dogfood features or do the required discovery work to provide feedback to Product. It is Product's responsibility to prioritize improvements or rebuild functionality in GitLab.

Dogfooding Antipatterns

An easy antipattern to fall into is to resolve your problem outside of what the product offers. Dogfooding is not:

  1. Building a bot outside of GitLab.
  2. Writing scripts that leverage the GitLab API (if the functionality is on our roadmap and could be shipped within the GitLab Project).
  3. Using a component of GitLab outside of the official installation methods.

Dogfooding Process

Follow the dogfooding process described in the Product Handbook when considering building a tool outside of GitLab.

GitLab Repositories

GitLab consists of many subprojects. A curated list of GitLab Repositories can be found at the GitLab Engineering Projects page.

When adding a repository please follow these steps:

  1. Ensure that the project is under the gitlab-org namespace for anything related to the application or under the gitlab-com namespace for anything strictly company related.
  2. Add the project to the list of GitLab Repositories
  3. Add an MIT license to the repository. It is easiest to simply copy-paste the MIT License verbatim from the gitlab repo.
  4. Add a section titled "Developer Certificate of Origin and License" to CONTRIBUTING.md in the repository. It is easiest to simply copy-paste the DCO + License section verbatim from the gitlab repo.
  5. Add any further relevant details to the Contribution Guide. See Contribution Example.
  6. Add a link to CONTRIBUTING.md from the project's README.md
  7. Add a CODEOWNERS file, to make it easy for contributors to figure out which teams are best suited to review their changes.
    • Use teams rather than individuals as owners, to make it self updating over time and resilient to people taking time off
    • You can scope ownership to subdirectories or individual files, but it should contain at the very least a top-level catch all for any new or non explicitly mentionned file.

When changing the settings in an existing repository, it's important to keep communication in mind. In addition to discussing the change in an issue and announcing it in relevant chat channels (e.g., #development), consider announcing the change during the Company Call. This is particularly important for changes to the GitLab repository.

Access Requests

GitLab consists of many different types of applications and resources.

When you require escalated permissions or privileges to a resource to conduct task(s), or support for creating resource(s) with specific endpoints, please submit an issue to the Access Requests Issue Tracker using the template provided.

Below is a short list of supported technologies:

Engineering Departments, Sections & Teams

Headcount planning

Before the beginning of each fiscal year, and at various check points throughout the year, we plan the size and shape of the Engineering and Product Management functions together to maintain symmetry.

The process should take place in a single artifact (usually a spreadsheet, current spreadsheet), and follow these steps:

  1. Product Management: Supplies headcount numbers for PMs and development groups proportional to our roadmap efforts
  2. Engineering: Supplies feedback to Product, headcount for management roles in the development department, and full plans for the Security, UX, Quality, and Infrastructure departments
  3. CEO: Supplies feedback to Engineering and Product, or gives final approval

Note: Support is part of the engineering function but is budgeted as 'cost of sales' instead of research and development. Headcount planning is done separately according to a different model.

Long Term Profitability Targets

The non support related departments within Engineering (Development, Infrastructure, Quality, Security, and UX) have an expense target of 20% as a percentage of revenue.

The Support target is 10% as a percentage of revenue.

Starting new teams

Our product offering is growing rapidly. Occasionally we start new teams. Backend teams should map to our product categories. Backend teams also map 1:1 to product managers.

A dedicated team needs certain skills and a minimum size to be successful. But that doesn't block us from taking on new work. This is how we iterate our team size and structure as a feature set grows:

  1. Existing Team: The existing PM schedules issues for most appropriate existing engineering team
    • If there is a second PM for this new feature, they work through the first PM to preserve the 1:1 interface
  2. Shared Manager Team: Dedicated engineer(s) are identified on existing teams and given a specialty
    • The manager must do double-duty
    • Their title can reflect both specialties of their engineers e.g. Engineering Manager, Distribution & Package
    • Even if temporary, managing two teams is a valuable career opportunity for a manager looking to develop director-level skills * Each specialty can have its own process, for example: Capitalized team label, Planning meetings, Standups
  3. New Dedicated Team:
    • Engineering Manager
    • Senior/Staff Engineer
    • Two approved fulltime vacancies
    • A dedicated PM

Team Page Template

## Vision

...

## Mission

...

## Team Members

The following people are permanent members of the [Blank] Team:

<table>
<thead>
<tr>
<th>Person</th>
<th>Role</th>
</tr>
</thead>
<tbody>
</tbody>
</table>


## Stable Counterparts

The following members of other functional teams are our stable counterparts:

<table>
<thead>
<tr>
<th>Person</th>
<th>Role</th>
</tr>
</thead>
<tbody>
</tbody>
</table>


## Hiring

This chart shows the progress we're making on hiring. Check out our
[jobs page](/jobs/) for current openings.

<script src="/javascripts/libs/moment.min.js" type="text/javascript" data-cookieconsent="ignore"></script>
<script src="/javascripts/libs/Chart.min.js" type="text/javascript" data-cookieconsent="ignore"></script>

<canvas id="Chart46625" width="400" height="200"></canvas>

<script data-cookieconsent="ignore">
window.addEventListener('load', function() {
  var ctx = document.getElementById('Chart46625').getContext('2d');

  var myChart = new Chart(ctx, {
    type: 'line',
      data: {
        datasets : [
          {
            label : 'people',
            fill: false,
            lineTension: 0,
            borderColor: 'red',
            data : [ { t: new Date('2019-11-22'), y: 0 }, ]
          },{
            label : 'vacancies',
            fill: false,
            lineTension: 0,
            borderColor: 'pink',
            data : [ { t: new Date('2019-11-22'), y: 0 }, ]
          }
       ]
    },
    options: {
      scales: {
        xAxes: [{
          type: 'time',
          time : {
            parser: 'YYYY-MM-DD'
          }
        }],
        yAxes: [{
          scaleLabel: {
            display: true,
            labelString: 'Headcount'
          }
        }]
      }
    }
  });
});
</script>





## Common Links

 * Issue Tracker
 * Slack Channel
 * ...

 ## How to work with us

 ...

Fast Boot Events

New teams may benefit from holding a Fast Boot event to help the jump start the team. During a Fast Boot, the entire team gets together in a physical location to bond and work alongside each other.

Mentorship and Coaching Programs

All levels of leadership at GitLab could benefit from external mentorship and coaching programs. To validate this hypothesis we are working on a small pilot program for 6 months with PlatoHQ and 7CTOs. After the program we will gather feedback before continuing or scaling the program out more broadly.

Collaboration

To maintain our rapid cadence of shipping a new release on the 22nd of every month, we must keep the barrier low to getting things done. Since our team is distributed around the world and therefore working at different times, we need to work in parallel and asynchronously as much as possible.

That also means that if you are implementing a new feature, you should feel empowered to work on the entire stack if it is most efficient for you to do so.

Nevertheless, there are features whose implementation requires knowledge that is outside the expertise of the developer or even the group/stage group. For these situations, we'll require the help of an expert in the feature's domain.

In order to figure out how to articulate this help, it is necessary to evaluate first the amount of work the feature will require from the expert.

If the feature only requires the expert's help at an early stage, for example designing and architecting the future solution, the approach will be slightly different. In this case, we would require the help of at least two experts in order to get a consensual agreement about the solution. Besides, they should be informed about the development status before the final solution is finished. This way, any discrepancy or architectural issue related to the current solution, will be brought up early.

Code Quality and Standards

We need to maintain code quality and standards. It's very important that you are familiar with the Development Guides in general, and the ones that relates to your group in particular:

Please remember that the only way to make code flexible is to make it as simple as possible:

Quality is everyone's responsibility

It is important to remember that quality is everyone's responsibility. Everything you merge to master should be production ready. Familiarize yourself with the definition of done.

Visualization Tools

Grafana

Kibana

Monitoring Tools

Prometheus

Sentry

Sitespeed.io

Error Budgets

We use SRE-like error budgets in OKRs to incentivize risk management and help make GitLab.com ready for mission critical customer workloads.

Each backend and frontend development team is responsible for not exceeding an allocated budget of 15 points each quarter. The severity of issues caused will impact their budget accordingly:

The Infrastructure team will perform attribution as part of the root cause analysis process and record the results in the OKRs page.

Engineering Proposed Initiatives

Engineering is the primary advocate for the performance, availability, and security of the GitLab project. Product Management prioritizes all initiatives, so everyone in the engineering function should participate in the Product Management prioritization process to ensure that our project stays ahead in these areas. The following list should provide some guidelines around the initiatives that each engineering team should advocate for during their release planning:

Rails by default, VueJS where it counts

Part of our engineering culture is to keep shipping so users and customers see significant new value added to GitLab.com or their self-managed instance. To support rapid development, we focus on Rails page views by default. When an area of the application sees significant usage, we typically rewrite those screens as a VueJS single page app backed by our API, in order to maintain the best qualitative experience and quantitative performance.

GraphQL first

When adding new functionality, we should use GraphQL where possible on the backend and the frontend. We have a long-term goal to use GraphQL everywhere because it lets us increase development speed, reduces dependencies between frontend and backend engineers, and gives us a single source of truth for application data.

Defaulting to GraphQL for new work means that the distance from that goal doesn't increase over time.

This does not override the importance of velocity: if something is significantly more work to ship using GraphQL, rather than extending an existing implementation (in a Rails controller or the REST API), we should not block ourselves on using GraphQL. Instead, we should ship the feature and create a follow-up issue to move that resource to GraphQL in future. That follow-up issue can be scheduled by the relevant Product Manager, in consultation with Engineering Managers, as with any other engineering proposed initiative.

Demos

Moved to a dedicated page.

Canary Testing

GitLab makes use of a 'Canary' stage. Production Canary is a series of servers running GitLab code in a production environment. The Canary stage contains code functional elements like web, container registry and git servers while sharing data elements such as sidekiq, database, and file storage with production. This allows UX code and most application logic code to be consumed by a smaller subset of users under real world scenarios before being made available to all users on GitLab.com.

The production Canary stage is forcibly enabled for all users visiting GitLab Inc. operated groups:

The Infrastructure department teams can globally disable use of production Canary when necessary. Individuals can also opt-out of using production Canary environments. However, opting-out does not include the aforementioned groups above.

To opt in/out, go to GitLab Version and move the toggle appropriately.

To verify that Canary is enabled, in the header, next to the GitLab logo will be a 'Next' icon, or use the performance bar (typing pb) in GitLab and watch out for the Canary icon next to the web server name.

Resources for Development

When using any of the resources listed below, some rules apply:

Google Cloud Platform (GCP)

Every team member has access to a common project on Google Cloud Platform. Please see the secure note with the name "Google Cloud Platform" in the shared vault in 1password for the credentials or further details on how to gain access.

Once in the console, you can spin up VM instances, Kubernetes clusters, etc. Where possible, please prefix the resource name with your name for easy identification (e.g. myname-k8s-cluster). Please remove any resources that you are not using, since the company is billed monthly. If you are unable to create a resource due to quota limits, file an issue on the Infrastructure issue tracker.

If your group needs to have its own GCP project, please use this issue template to request one. Your group may already have a project which can be found on this list of group GCP projects.

If you encounter the following error when creating a new GKE cluster, this indicates that we cannot create more clusters within that network. Please ask in #kubernetes for team members to delete unused clusters, or alternatively create your cluster in a different network.

The network "default" does not have available private IP space in 10.0.0.0/8

Digital Ocean (DO)

Every team member has access to the dev-resources project which allows everyone to create and delete machines on demand.

Amazon Web Services (AWS)

In general, most team members do not have access to AWS accounts. In case you need an AWS resource, file an issue on the Infrastructure issue tracker. Please supply the details on what type of access you need.

DevOps Slack Channels

There are primarily two Slack channels which developers may be called upon to assist the production team when something appears to be amiss with GitLab.com:

  1. #backend: For backend-related issues (e.g. error 500s, high database load, etc.)
  2. #frontend: For frontend-related issues (e.g. JavaScript errors, buttons not working, etc.)

Treat questions or requests from production team for immediate urgency with high priority.