Engineering

Communication

At GitLab, we communicate in the open whenever possible. We want each discussion point to be viewable by as many people as possible. This allows anyone who is able to provide a valuable response faster, where something like an email or a direct message may require waiting on somebody to return from vacation, for example. The following communication channels are listed in the order in which they should be considered.

On this page

The Importance of Velocity

Velocity over predictability

We optimize for shipping a high volume of user/customer value with each release. We do want to ship multiple major features in every monthly release of GitLab. However, we do not strive for predictability over velocity. As such, we eschew heavyweight processes like detailed story point estimation by the whole team in favor of lightweight measurements of throughput like the number of merge requests that were included or rough estimates by single team members.

There is variance in how much time an issue will take versus what you estimated. This variance causes unpredictability. If you want close to 100% predictability you have to take two measures:

  1. Invest more time in estimation to reduce that variance. The time spend estimating things could otherwise be used to create features.
  2. Leave a reserve of time with unscheduled work so you can accommodate the variance. According to Parkinson's law the work expands so as to fill the time available for its completion. This means that we're not adhering to our iteration value and that for the next cycle our estimates for comparable features will be larger.

Both measures reduce the overall velocity of shipping features. The way to prevent this is to accept that we don't want perfect predictability. Just like our OKRs are so ambitious that we expect to reach about 70% of the goal this is fine for shipping planned features too.

Note: This does not mean we place zero value on predictability. We just optimize for velocity first.

GitLab Repositories

GitLab consists of many subprojects. A curated list of GitLab Repositories can be found at the GitLab Engineering Projects page.

When adding a repository please follow these steps:

  1. Ensure that the project is under the gitlab-org namespace for anything related to the application or under the gitlab-com namespace for anything strictly company related.
  2. Add the project to the list of GitLab Repositories
  3. Add an MIT license to the repository. It is easiest to simply copy-paste the MIT License verbatim from the gitlab-ce repo.
  4. Add a section titled "Developer Certificate of Origin and License" to CONTRIBUTING.md in the repository. It is easiest to simply copy-paste the DCO + License section verbatim from the gitlab-ce repo.
  5. Add any further relevant details to the Contribution Guide. See Contribution Example.
  6. Add a link to CONTRIBUTING.md from the project's README.md

Access Requests

GitLab consists of many different type of applications and resources.

When you require escalated permissions or privileges to a resource to conduct task(s), or support for creating resource(s) with specific endpoints, please submit an issue to the Access Requests Issue Tracker using the template provided.

Below is a short list of supported technologies:

Engineering Departments & Teams

Starting new teams

Our product offering is growing rapidly. Occasionally we start new teams. Backend teams should map to our product categories. Backend teams also map 1:1 to product managers.

A dedicated team needs certain skills and a minimum size to be successful. But that doesn't block us from taking on new work. This is how we iterate our team size and structure as a feature set grows:

  1. Existing Team: The existing PM schedules issues for most appropriate existing engineering team
    • If there is a second PM for this new feature, they work through the first PM to preserve the 1:1 interface
  2. Shared Manager Team: Dedicated engineer(s) are identifed on existing teams and given a specialty
    • The manager must do double-duty
    • Their title can reflect both specialties of their engineers e.g. Engineering Manager, Distribution & Packaging
    • Even if temporary, managing two teams is a valuable career opportunity for a manager looking to develop director-level skills * Each specialty can have its own process, for example: Capitalized team label, Planning meetings, Standups
  3. New Dedicated Team:
    • Engineering Manager
    • Senior/Staff Engineer
    • Two approved fulltime vacancies
    • A dedicated PM

Team Page Template

## Vision

...

## Mission

...

## Team Members

The following people are permanent members of the [Blank] Team:

<%= direct_team(manager_role: 'Engineering Manager, [Blank]') %>

## Stable Counterparts

The following members of other functional teams are our stable counterparts:

<%= stable_counterparts(role_regexp: /[,&] Blank/, direct_manager_role: 'Engineering Manager, [Blank]') %>

## Common Links

 * Issue Tracker
 * Slack Channel
 * ...

 ## How to work with us

 ...

Collaboration

To maintain our rapid cadence of shipping a new release every month, we must keep the barrier low to getting things done. Since our team is distributed around the world and therefore working at different times, we need to work in parallel and asynchronously as much as possible.

That also means that if you are implementing a new feature, you should feel empowered to work on the entire stack if it is most efficient for you to do so.

Code Quality and Standards

We need to maintain code quality and standards. It's very important that you are familiar with the Development Guides in general, and the ones that relates to your group in particular:

It is important to remember that quality is everyone's responsibility. Everything you merge to master should be production ready. Familiarize yourself with the definition of done.

Error Budgets

In Q3 of 2018 we are trying SRE-like error budgets in OKRs to incentivize risk management help and make GitLab.com ready for mission critical customer workloads.

Each backend and frontend development team will be responsible for preserving 100 points (or percent) of their budget this quarter. The severity of issues caused will subtract from their budget accordingly:

The Infrastructure team will perform attribution as part of the root cause analysis process and record the results in the OKRs page.

Engineering Proposed Initiatives

Engineering is the primary advocate for the performance, availability, and security of the GitLab project. Product Management prioritizes all initiatives, so everyone in the engineering function should participate in the Product Management prioritization process to ensure that our project stays ahead in these areas. The following list should provide some guidelines around the initiatives that each engineering team should advocate for during their release planning:

Rails by default, VueJS where it counts

Part of our engineering culture is to keep shipping so users and customers see significant new value added to GitLab.com or their self-managed instance. To support rapid development, we focus on Rails page views by default. When an area of the application sees significant usage, we typically rewrite those screens as a VueJS single page app backed by our API, in order to maintain the best qualitative experience and quantitative performance.

Demos

The idea that working software is the primary measure of progress is one of the principles of agile software development. Demoing gets more eyes on the project to uncover bugs and reveal ambiguities in the product requirements. It's also a transparent and lightweight way to provide status to the rest of the organization. Developers should demo at least once a week with product managers present. Demo meetings should be kept to 30 minutes or less. The emphasis should be on the product requirements or acceptance criteria and less on the implementation details.

Canary Testing

GitLab makes use of a 'Canary' stage, a series of servers to test the GitLab code base on prior to production deployment. The Canary stage contains code functional elements like web and git servers while sharing data elements such as sidekiq, database, and file storage with production. This allows UX code and most application logic code to be user tested under real world scenarios before being deployed.

The Canary stage is available for usage by enabling a cookie attribute of gitlab_canary when accessing gitlab.com. All engineers are encouraged to enable this flag and perform normal daily activities in order to spot issues and problems before deployment to the production environment. In order to facilitate the enabling of the canary cookie, a short javascript snipped is provided below to bookmark that toggles between states.

javascript:void((function(d){document.cookie='gitlab_canary=' + (document.cookie.indexOf('gitlab_canary=true') >= 0 ?  'false' : 'true') + ';domain=.gitlab.com;path=/;expires=' + new Date(Date.now() + 31536000000).toUTCString(); location.reload();})(document));

Resources for Development

When using any of the resources listed below, some rules apply:

Google Cloud Platform (GCP)

Every team member has access to a common project on Google Cloud Platform. Please see the secure note with the name "Google Cloud Platform" in the shared vault in 1password for the credentials or further details on how to gain access.

Once in the console, you can spin up VM instances, Kubernetes clusters, etc. Please remove any resources that you are not using, since the company is billed monthly. If you are unable to create a resource due to quota limits, file an issue on the Infrastructure issue tracker.

Digital Ocean (DO)

Every team member has access to the dev-resources project which allows everyone to create and delete machines on demand.

Amazon Web Services (AWS)

In general, most team members do not have access to AWS accounts. In case you need an AWS resource, file an issue on the Infrastructure issue tracker. Please supply the details on what type of access you need.

DevOps Slack Channels

There are primarily two Slack channels which developers may be called upon to assist the production team when something appears to be amiss with GitLab.com:

  1. #backend: For backend-related issues (e.g. error 500s, high database load, etc.)
  2. #frontend: For frontend-related issues (e.g. JavaScript errors, buttons not working, etc.)

Treat questions or requests from production team for immediate urgency with high priority.

Reaction Rotation

See the Reaction description.

Release Managers

See the Release Management description.