Engineering Workflow

  1. You are here:
  2. Team Handbook
  3. Engineering
  4. Engineering Workflow

This document explains the workflow for anyone working with issues in GitLab Inc. For the workflow that applies to everyone please see PROCESS.md.

On this page

GitLab Flow

Products at GitLab are built using the GitLab Flow.

Broken master

If you notice that pipelines for the master branch of GitLab CE or EE are failing (red) or broken (green as a false positive), fixing this takes priority over everything else development related, since everything we do while tests are broken may break existing functionality, or introduce new bugs and security issues.

As a Developer

  1. Look for an issue labelled ~"broken master" that would track the failure
  2. If it doesn't exist, create it, ping the relevant people, and post it in #development so that other developers are aware of the problem and can help
  3. If you have the time, assign it to yourself and start working on it
  4. Reply to the automatic "Pipeline failed" notifications in #development by mentioning the issue
  5. If the problem isn't fixed within a few hours, @mention the relevant Engineering Leads and CTO in the issue and on Slack, so that resources can be assigned to fix it as quickly as possible.

As a Maintainer

It's ok to merge a merge request with a failing pipeline if the following conditions are met:

  1. The failure also happens on master, and the failing specs are not directly related to functionality touched by the merge request
  2. There is an issue labelled ~"broken master" for it, see the "As a developer" steps above for more details

Before merging, it's a good practice to mention that the failure happens in master, and to post a reference to the issue. For instance:

Failure in <JOB_URL> happens in `master` and is being worked on in #XYZ, merging.

Security Issues

Security issues are managed and prioritized by the security team. If you are assigned to work on a security issue in a milestone, you need to follow these steps:

  1. Do not push code relating to the fix to GitLab.com. The changes should be kept confidential until the release, and there is no way to make merge requests on GitLab.com confidential at the moment.
  2. Instead, work against the equivalent project on dev.gitlab.org. For instance, GitLab Community Edition on dev is at gitlab/gitlabhq.
  3. Create your merge request from the most recent security branch. For example, if the current release is 23.5, then use the security-23-5 branch as the target for your merge request.

  4. If no security branch exists, create one from the latest stable branch. For the previous example, you can run (assuming dev is the remote):

     git checkout -b security-23-5 origin/23-5-stable
 git push -u dev HEAD
  5. Double link the issue to the merge request on dev, and the merge request on dev to the issue.
  6. Most security issues will need backports to the previous two monthly releases. For example, if the current release is 23.5, then we would also need MRs to security-23-4 and security-23-3.
    1. Note: you do not have to create the backport merge requests until the merge request has been reviewed and is ready to merge, if you would rather wait.
  7. Check and add a note in the security issue you're working on listing which GitLab versions does it affects. For example:
    • GitLab CE and EE 8.9.0 - 9.5.10
    • GitLab CE and EE 10.0.0 - 10.1.5
    • GitLab CE and EE 10.2.0 - 10.2.5 These will be included in the upcoming blog posts.
  8. Security patches will eventually be merged into master using a patch set from the most recent stable branch. After the merge requests for the stable branches have been merged, the release manager will ping you if the changes don't apply cleanly to master or pass all tests.
  9. If an issue exists for the target security patch release, link the merge requests with all the backports and relevant issue to that as well.

If you find a security issue in GitLab, create a confidential issue mentioning the relevant security and engineering managers, and post about it in #security.

If you accidentally push security commits to GitLab.com, we recommend that you:

  1. Delete the relevant branch ASAP
  2. Inform a release manager in #releases. It may be possible to execute a garbage collection (via the Housekeeping task in the repository settings) to remove the commits.

For more information on how the entire process works for security releases, see the documentation on security releases.

Basics

  1. Start working on an issue you’re assigned to. If you’re not assigned to any issue, find the issue with the highest priority you can work on, by relevant label. You can use this query, which sorts by priority for the started milestones, and filter by the label for your team.
  2. If you need to schedule something or prioritize it, apply the appropriate labels (see Scheduling issues).
  3. If you are working on an issue that touches on areas outside of your expertise, be sure to mention someone in the other group(s) as you soon as you start working on it. This allows others to give you early feedback, which should save you time in the long run.
  4. You are responsible for the issues assigned to you. This means it has to ship with the milestone it's associated with. If you are not able to do this, you have to communicate it early to your manager and other stakeholders (e.g. the product manager, other engineers working on dependent issues). In teams, the team is responsible for this (see Working in Teams). If you are uncertain, err on the side of overcommunication. It's always better to communicate doubts than to wait.
  5. You (and your team, if applicable) are responsible for:
  6. Once a release candidate has been deployed to the staging environment, please verify that your changes work as intended. We have seen issues where bugs did not appear in development but showed in production (e.g. due to CE-EE merge issues).

For general guidelines about issues and merge requests, be sure to read the GitLab Workflow.

Working in Teams

For larger issues or issues that contain many different moving parts, you'll be likely working in a team. This team will typically consist of a backend developer, a frontend developer, a UX designer and a product manager.

  1. Teams have a shared responsibility to ship the issue in the planned release.
    1. If the team suspects that they might not be able to ship something in time, the team should escalate / inform others as soon as possible. A good start is informing your manager.
    2. It's generally preferable to ship a smaller iteration of an issue, than ship something a release later.
  2. Consider starting a Slack channel for a new team, but remember to write all relevant information in the related issue(s). You don't want to have to read up on two threads, rather than only one, and Slack channels are not open to the greater GitLab community.

Convention over Configuration

Avoid adding configuration values in the application settings or in gitlab.yml. Only add configuration if it is absolutely necessary. If you find yourself adding parameters to tune specific features, stop and consider how this can be avoided. Are they values really necessary? Could constants be used that work across the board? Could values be determined automatically? See Convention over Configuration for more discussion.

Choosing Something to Work On

Start working on things with the highest priority in the current milestone. The priority of items are defined under labels in the repository, but you are able to sort by priority.

After sorting by priority, choose something that you’re able to tackle and falls under your responsibility. That means that if you’re a frontend developer, you work on something with the label frontend.

To filter very precisely, you could filter all issues for:

Use this link to quickly set the above parameters. You'll still need to filter by the label for your own team.

If you’re in doubt about what to work on, ask your lead. They will be able to tell you.

Triaging and Reviewing Code from the rest of the Community

It's every developers' responsibilities to triage and review code contributed by the rest of the community, and work with them to get it ready for production.

Merge requests from the rest of the community should be labeled with the Community Contribution label.

When evaluating a merge request from the community, please ensure that a relevant PM is aware of the pending MR by mentioning them.

This should be to be part of your daily routine. For instance, every morning you could triage new merge requests from the rest of the community that are not yet labeled Community Contribution and either review them or ask a relevant person to review it.

Make sure to follow our Code Review Guidelines.

Workflow Labels

Labels are described in our Contribution guide.

Working with GitLab.com

GitLab.com is a very large instance of GitLab Enterprise Edition. It runs release candidates for new releases, and sees a lot of issues because of the amount of traffic it gets. There are several internal tools available for developers at GitLab to get data about what's happening in the production system:

Performance Data

There is extensive monitoring publicly available for GitLab.com. For more on this and related tools, see the monitoring handbook.

More details on GitLab Profiler are also found in the monitoring performance handbook.

Error Reporting

Feature Flags

If you've built feature flags into your code, be sure to read about how to use the feature flag to test a feature on GitLab.com.

Scheduling Issues

GitLab Inc has to be selective in working on particular issues. We have a limited capacity to work on new things. Therefore, we have to schedule issues carefully.

Product Managers are responsible for scheduling all issues in their respective product areas, including features, bugs, and tech debt. Product managers alone determine the prioritization. The UX Lead and Engineering Leads are responsible for allocating people making sure things are done on time. Product Managers are not responsible for these activities, they are not project managers.

Direction issues are the big, prioritized new features for each release. They are limited to a small number per release so that we have plenty of capacity to work on other important issues, bug fixes, etc.

If you want to schedule an Accepting Merge Requests issue, please remove the label first.

Any scheduled issue should have a team label assigned, and at least one type label.

Requesting Something to be Scheduled

To request scheduling an issue, ask the responsible product manager

We have many more requests for great features than we have capacity to work on. There is a good chance we’ll not be able to work on something. Make sure the appropriate labels (such as customer) are applied so every issue is given the priority it deserves.

Process Improvement

There is an informal scheduling process discussion in the #scheduling Slack channel. Anyone can join and suggest improvements to our scheduling process.

Product Development Timeline

Teams (Product, UX, Engineering) continually work on issues according to their respective workflows. There is no specified process whereby a particular person should be working on a set of issues in a given time period. However, there are specific deadlines that should inform team workflows and prioritization. Suppose we are talking about milestone m that will be shipped in month M (on the 22nd). We have the following deadlines:

Refer to release post due dates for additional deadlines.

Note that release timelines are overlapping. For example, when a release is shipped to production on the 22nd, the scope for the following release has already been established earlier in that same month.

An in-scope issue already has a docs starter blurb (written by the PM) by the time engineers start development. Engineers should create and merge in the updated docs as part of completing an issue by the 7th. PMs revise the starter docs blurbs in the release post appropriately before the release post is published.

Refer to Feature freeze on the 7th for the release on the 22nd for further timeline details of code releases, including major/minor version releases, as well as patch releases.

Updating Issues Throughout Development

Team members use labels to track issues throughout development. This gives visibility to other developers, product managers, and designers, so that they can adjust their plans during a monthly iteration. An issue should follow these stages:

Kickoff

At the beginning of each release, we have a kickoff meeting, publicly livestreamed to YouTube. In the call, the Product Development team (PMs, UX designers, and Engineers) communicate with the rest of the organization which issues are in scope for the upcoming release. The call is structured by product area with each PM leading their part of the call.

The notes are available in a publicly-accessible Google doc. Refer to the doc for details on viewing the livestream.

Retrospective

After each release, we have a retrospective meeting, publicly livestreamed to YouTube. We discuss what went well, what went wrong, and what we can improve for the next release.

In each retrospective we choose a low number (1-3) of high-impact improvements that can be realistically delivered before the next release. A single champion is identified to deliver each improvement. A Champion then follows up on their task to completion. Before the start of the next retrospective, champions are expected to resolve their action items and update the retrospective document on their improvement tasks. And we talk about the status of each improvement at the start of the following retrospective.

The notes are available in a publicly-accessible Google doc. Refer to the doc for details on viewing the livestream.

Kickoff and Retrospective Livestream Instructions

Before the meeting starts, remind people who plan to speak to join the Google Hangout earlier, since there is a 50 user limit.

Several minutes before the scheduled meeting time, follow the livestreaming instructions to start a Google Hangout using the Now setting. Paste the Google Hangout invite link in the Google doc.

At the scheduled meeting time, start broadcasting live to YouTube. Begin the meeting.

Use Group Labels and Group Milestones

When working in GitLab (and in particular, the GitLab.org group), use group labels and group milestones as much as you can. It is easier to plan issues and merge requests at the group level, and exposes ideas across projects more naturally. If you have a project label, you can promote it to a group milestone. This will merge all project labels with the same name into the one group label. The same is true for promoting group milestones.

Technical debt

We definitely don't want our technical debt to grow faster than our code base. To prevent this from happening we should consider not only the impact of the technical debt but also a contagion. How big and how fast this problem is going to be over time? Is it likely a bad piece of code will be copy-pasted for a future feature? In the end, the amount of resources available is always less than amount of technical debt to address.

To help with prioritization and decision-making process here, we recommend thinking about contagion as an interest rate of the technical debt. There is a great comment from the internet about it:

You wouldn't pay off your $50k student loan before first paying off your $5k credit card and it's because of the high interest rate. The best debt to pay off first is one that has the highest loan payment to recurring payment reduction ratio, i.e. the one that reduces your overall debt payments the most, and that is usually the loan with the highest interest rate.

Edit this page — please contribute. Creative Commons License