Build

Team responsibility

Build is about how we ship GitLab, and making sure everyone can easily install, update and maintain GitLab.

This means:

Projects

Name Location Description
Omnibus Gitlab gitlab-org/omnibus-gitlab Build Omnibus packages with HA support for LTS versions of all major Linux operating systems such as Ubuntu, Debian, CentOS/RHEL, OpenSUSE, SLES
Docker GitLab image gitlab-org/omnibus-gitlab/docker Build Docker images for GitLab CE/EE based on the omnibus-gitlab package
AWS images AWS marketplace AWS image based on the omnibus-gitlab package
Azure images Azure marketplace Azure image based on the omnibus-gitlab package
Kubernetes helm charts charts/charts.gitlab.io Official Helm Charts Application definitions for Kubernetes Helm based on the omnibus-gitlab package
Redhat Openshift Openshift template Template for Openshift Origin based on the omnibus-gitlab package
Mesosphere DC/OS package Universe repository Package for Mesosphere DC/OS based on omnibus-gitlab package
GitLab PCF tile gitlab.com/gitlab-pivotal One click installation of GitLab in Pivotal Cloud Foundry based on omnibus-gitlab package
GitLab Terraform configuration gitlab-terraform Terraform configuration for various cloud providers
Omnibus GitLab Builder GitLab Omnibus Builder Create environment containing build dependencies for the omnibus-gitlab package

How to work with Build

Everything that is done in GitLab will end up in the packages that are shipped to users. While that sounds like the last link in the chain, it is one of the most important ones. This means that informing the Build team of a change in an early stage is crucial for releasing your feature. While last minute changes are inevitable and can happen, we should strive to avoid them.

We expect every team to reach out to the Build team before scheduling a feature in an upcoming release in the following cases:

To sum up the above list:

If you need to do install, update, make, mkdir, mv, cp, chown, chmod, compilation or configuration change in any part of GitLab stack, you need to reach out to the Build team for opinion as early as possible.

This will allow us to schedule appropriately the changes (if any) we have to make to the packages.

If a change is reported late in the release cycle or not reported at all, your feature/change might not be shipped within the release.

If you have any doubt whether your change will have an impact on the Build team, don't hesitate to ping us in your issue and we'll gladly help.

Internal team training

Every Build team member is responsible for creating a training session for the rest of the team. These trainings will be recorded and available to the whole team.

Purpose

The purpose of team training is to introduce the work done to the rest of your team. It also allows the rest of the team to easily transition into new features and projects and prepare them for maintenance.

Training should be:

Training should not be:

Simply put, the training is a summation of: notes taken in issues during development, programming challenges, high level overview of written documentation. Your team member should be able to take over the maintenance or build on top of your feature with less effort after they have been part of the training.

Note Do not shy away from being technical in your training. You can ask yourself: What would have been useful for me when I started working on this task? What would have helped me be more efficient?

Efficiency of the training

In order to see whether the training is efficient, Build lead will rotate team members on projects where training was done. For example, if the feature requires regular releases, the person who gave the training will be considered a tutor. Different team member will follow the training and documentation and will ask the original maintainer for help. The new person responsible is now also responsible for improving the feature. They are now also responsible of training other team members.

FAQ

Q: Isn't this double work? A: No. The training should be prepared while documenting the task.

Q: Won't this slow me down? A: At the beginning, possibly. However, every hour of the training given will multiple the value of it by the amount of team members.

Q: Isn't it more useful to let the team check out the docs and ask questions? A: In an ideal world, possibly. However, everyone has a lot of tasks assigned and they might not be able to go through the docs until they need to do something. This might be months later and you, as a person who would give the training, might not be able help efficiently anymore.

Public by default

All work carried out by the Build team is public. Some exceptions apply:

If you are unsure whether something needs to remain private, check with the team lead.

Working on dev.gitlab.org

Some of the team work is carried out on our development server at dev.gitlab.org. Infrastructure overview document lists the reasons.

Unless your work is related to the security, all other work is carried out in projects on GitLab.com.

Cloud Images

The process documenting the steps necessary to update the GitLab images available on the various cloud providers is detailed on our Cloud Image Process page.

Infrastructure

As part of the team tasks, team has responsibility towards the following nodes:

dev.gitlab.org

Every day at 1:30 UTC, a nightly build gets triggered on dev.gitlab.org. The cron trigger times are currently defined at the scheduled pipeline page on dev.gitlab.org.

Every day at 7:20 UTC, the nightly CE packages gets automatically deployed on dev.gitlab.org. Any errors in the install process will be logged in Sentry. Slack notifications will appear in #dev-gitlab. The cron task is currently defined in dev.gitlab.org role.

Maintenance tasks

Requirements:

Teams responsibility is to make sure that the GitLab instance on this server is operational. The omnibus-gitlab package on this server is a stock package with required configuration to keep it operational. Regular omnibus-gitlab commands can be used on this node. If, for some reason, you need to apply a change to /etc/gitlab/gitlab.rb, this change needs to be introduced in the dev-gitlab-org role.

If you do not have access to this repository, but you need to do a hot-patch or configuration testing, the following steps can be performed:

Improvements

omnibus-builder-runners-manager.gitlab.org

GitLab CI runner manager is responsible for creating build machines for package builds. This node configuration is managed by cookbook-gitlab-runner. Configuration values are stored in the corresponding vault.

Currently, the version of GitLab CI runner is locked. We aim to be close to the current version of runner in order to get the fixes that we need without getting into issues that could cause a failure. These failures could prevent the release from going out so be careful with unnecessary changes on this node.

The runner manager is configured to use two docker machine drivers, "digitalocean" and "scaleway".

The former boots up machines in a Digital Ocean account for package and Docker image builds. The latter boots up machines in a Scaleway account for Raspberry Pi (arm platform) builds.

Maintenance tasks

Requirements:

When the version of GitLab CI runner needs to be changed:

When you notice that the builds are pending on our dev.gitlab.org project, it is possible that the number of failed machines is high. Failed machines prevent the runner manager from starting up new machines and this can slow down or even block the release. To resolve this:

packages.gitlab.com

At this moment, Build team is only the user of packages.gitlab.com. Release packages are served to our users and customers from our CI on dev.gitlab.org.

The duties for this server are yet to be defined with the Production team.

Given that the package server is currently deployed on our own infrastructure, from an omnibus type package, if Production requires help the team should do a best effort to help trough any issues.