Engineering

Communication

On this page

GitLab Repositories

GitLab consists of many subprojects. A curated list of GitLab Repositories can be found at the GitLab Engineering Projects page.

When adding a repository please follow these steps:

  1. Ensure that the project is under the gitlab-org namespace for anything related to the application or under the gitlab-com namespace for anything strictly company related.
  2. Add the project to the list of GitLab Repositories
  3. Add an MIT license to the repository. It is easiest to simply copy-paste the MIT License verbatim from the gitlab-ce repo.
  4. Add a section titled "Developer Certificate of Origin and License" to CONTRIBUTING.md in the repository. It is easiest to simply copy-paste the DCO + License section verbatim from the gitlab-ce repo.
  5. Add any further relevant details to the Contribution Guide. See Contribution Example.
  6. Add a link to CONTRIBUTING.md from the project's README.md

Engineering Departments & Teams

Starting new teams

Our product offering is growing rapidly. Occasionally we start new teams. Backend teams should map to our product categories. Backend teams also map 1:1 to product managers.

A dedicated team needs certain skills and a minimum size to be successful. But that doesn't block us from taking on new work. This is how we iterate our team size and structure as a feature set grows:

  1. Existing Team: The existing PM schedules issues for most appropriate existing engineering team
    • If there is a second PM for this new feature, they work through the first PM to preserve the 1:1 interface
  2. Shared Manager Team: Dedicated engineer(s) are identifed on existing teams and given a specialty
    • The manager must do double-duty
    • Their title can reflect both specialties of their engineers e.g. Engineering Manager, Distribution & Packaging
    • Even if temporary, managing two teams is a valuable career opportunity for a manager looking to develop director-level skills * Each specialty can have its own process, for example: Capitalized team label, Planning meetings, Standups
  3. New Dedicated Team:
    • Engineering Manager
    • Senior/Staff Engineer
    • Two approved fulltime vacancies
    • A dedicated PM

Collaboration

To maintain our rapid cadence of shipping a new release every month, we must keep the barrier low to getting things done. Since our team is distributed around the world and therefore working at different times, we need to work in parallel and asynchronously as much as possible.

That also means that if you are implementing a new feature, you should feel empowered to work on the entire stack if it is most efficient for you to do so.

Code Quality and Standards

We need to maintain code quality and standards. It's very important that you are familiar with the Development Guides in general, and the ones that relates to your group in particular:

It is important to remember that quality is everyone's responsibility. Everything you merge to master should be production ready. Familiarize yourself with the definition of done.

Error Budgets

In Q3 of 2018 we are trying SRE-like error budgets in OKRs to incentivize risk management help and make GitLab.com ready for mission critical customer workloads.

Each backend and frontend development team will be responsible for preserving 100 points (or percent) of their budget this quarter. The severity of issues caused will subtract from their budget accordingly:

The Infrastructure team will perform attribution as part of the post-mortem process and record the results in the OKRs page.

Engineering Led Initiatives

Engineering is the primary advocate for the performance, availability, and security of the GitLab project. Everyone in the engineering function should participate in the Product Management prioritization process to ensure that our project stays ahead in these areas. The following list should provide some guidelines around the initiatives that each engineering team should advocate for during their release planning:

Rails by default, VueJS where it counts

Part of our engineering culture is to keep shipping so users and customers see significant new value added to GitLab.com or their self-managed instance. To support rapid development, we focus on Rails page views by default. When an area of the application sees significant usage, we typically rewrite those screens as a VueJS single page app backed by our API, in order to maintain the best qualitative experience and quantitative performance.

Velocity over predictability

We're optimize for shipping a high volume of user/customer value with each release. We do want to ship multiple major features in every monthly release of GitLab. However, we do not strive for predictability over velocity. As such, we eschew heavyweight processes like detailed story point estimation by the whole team in favor of lightweight measurements of throughput like the number of merge requests that were included or rough estimates by single team members.

There is variance in how much time an issue will take versus what you estimated. This variance causes unpredictability. If you want close to 100% predictability you have to take two measures:

  1. Invest more time in estimation to reduce that variance. The time spend estimating things could otherwise be used to create features.
  2. Leave a reserve of time with unscheduled work so you can accommodate the variance. According to Parkinson's law the work expands so as to fill the time available for its completion. This means that we're not adhering to our iteration value and that for the next cycle our estimates for comparable features will be larger.

Both measures reduce the overall velocity of shipping features. The way to prevent this is to accept that we don't want perfect predictability. Just like our OKRs are so ambitious that we expect to reach about 70% of the goal this is fine for shipping planned features too.

Demos

The idea that working software is the primary measure of progress is one of the principles of agile software development. Demoing gets more eyes on the project to uncover bugs and reveal ambiguities in the product requirements. It's also a transparent and lightweight way to provide status to the rest of the organization. Developers should demo at least once a week with product managers present. Demo meetings should be kept to 30 minutes or less. The emphasis should be on the product requirements or acceptance criteria and less on the implementation details.

Canary Testing

GitLab makes use of a 'Canary' environment, a series of servers to test the GitLab code base on prior to production deployment. The Canary environment contains code functional elements like web and git servers while sharing data elements such as sidekiq, database, and file storage with production. This allows UX code and most application logic code to be user tested under real world scenarios before being deployed.

The Canary environment is available for usage by enabling a cookie attribute of gitlab_canary when accessing gitlab.com. All engineers are encouraged to enable this flag and perform normal daily activities in order to spot issues and problems before deployment to the production environment. In order to facilitate the enabling of the canary cookie, a short javascript snipped is provided below to bookmark that toggles between states.

javascript:void((function(d){document.cookie='gitlab_canary=' + (document.cookie.indexOf('gitlab_canary=true') >= 0 ?  'false' : 'true') + ';domain=.gitlab.com;path=/;expires=' + new Date(Date.now() + 31536000000).toUTCString(); location.reload();})(document));

Code Reviews

Code reviews are mandatory for every merge request, you should get familiar and follow our Code Review Guidelines.

These guidelines also describe who would need to review, approve and merge your merge request.

Reviewer

All GitLab developers can, and are encouraged to, perform code review on merge requests of colleagues and community contributors. If you want to review merge requests, you can wait until someone assigns you one, but you are also more than welcome to browse the list of open merge requests and leave any feedback or questions you may have.

To let other developers know that you are interested in performing code reviews, you can add this to your team page entry by editing data/team.yml.

You can find someone to review your own merge requests by looking on the team page, or on the list of GitLab Engineering Projects, both of which are fed by data/team.yml.

You can also help community contributors get their merge requests ready, by becoming a Merge Request Coach.

Note that while all developers can review all merge requests, the ability to accept merge requests is restricted to maintainers.

Maintainer

Maintainers are GitLab developers who are experts at code review, know the GitLab product and code base very well, and are empowered to accept merge requests in one or several GitLab Engineering Projects.

Every project has at least one maintainer, but most have multiple, and some projects (like GitLab CE and GitLab EE) have separate maintainers for frontend and backend.

Great developers are often also great reviewers, but code review is a skill in and of itself, and not every developer, no matter their seniority, will have had the same opportunities to hone that skill. It's also important to note that a big part of being a good maintainer comes from knowing the existing product and code base extremely well, which lets them spot inconsistencies, edge cases, or non-obvious interactions with other features that would otherwise be missed easily.

To protect and ensure the quality of the code base and the product as a whole, people become maintainers only once they have convincingly demonstrated that their reviewing skills are at a comparable level to those of existing maintainers.

As with regular reviewers, maintainers can be found on the team page, or on the list of GitLab Engineering Projects.

How to become a maintainer

As a reviewer, a great way to improve your reviewing skills is to participate in MRs. Add your review notes, pass them on to maintainers, and follow the conversation until the MR is closed. If a comment doesn't make sense to you, ask the commenter to explain further. If you missed something in your review, figure out why you didn't see it, and note it down for next time.

Reviewers need to make the case for their own maintainership status.

When the MR's that they have reviewed usually make it through maintainer review without significant comments, they can:

If the existing maintainers do not have significant objections, and if at least half of them agree that the reviewer is indeed ready, we've got ourselves a new maintainer!

The existing maintainers will also raise any areas for growth on the merge request. If there are many gaps, the reviewer will need to address these before asking for reconsideration.

If you'd like to work towards becoming a maintainer, discuss it in your regular 1:1 meetings with your manager. They will help you to identify areas to work on before following the process above.

Resources for Development

When using any of the resources listed below, some rules apply:

Google Cloud Platform (GCP)

Every team member has access to a common project on Google Cloud Platform. Please see the secure note with the name "Google Cloud Platform" in the shared vault in 1password for the credentials or further details on how to gain access.

Once in the console, you can spin up VM instances, Kubernetes clusters, etc. Please remove any resources that you are not using, since the company is billed monthly. If you are unable to create a resource due to quota limits, file an issue on the Infrastructure issue tracker.

Digital Ocean (DO)

Every team member has access to the dev-resources project which allows everyone to create and delete machines on demand.

Amazon Web Services (AWS)

In general, most team members do not have access to AWS accounts. In case you need an AWS resource, file an issue on the Infrastructure issue tracker. Please supply the details on what type of access you need.

DevOps Slack Channels

There are primarily two Slack channels which developers may be called upon to assist the production team when something appears to be amiss with GitLab.com:

  1. #backend: For backend-related issues (e.g. error 500s, high database load, etc.)
  2. #frontend: For frontend-related issues (e.g. JavaScript errors, buttons not working, etc.)

Treat questions or requests from production team for immediate urgency with high priority.

Reaction Rotation

See the Reaction description.

Release Managers

See the Release Management description.