Distribution ensures the experience of installing and updating GitLab is easy and safe for everyone.
This means they make sure that:
Distribution label.| Name | Location | Description |
|---|---|---|
| Omnibus GitLab | gitlab-org/omnibus-gitlab | Build Omnibus packages with HA support for LTS versions of all major Linux operating systems such as Ubuntu, Debian, CentOS/RHEL, OpenSUSE, SLES |
| Docker GitLab image | gitlab-org/omnibus-gitlab/docker | Build Docker images for GitLab CE/EE based on the omnibus-gitlab package |
| AWS images | AWS marketplace | AWS image based on the omnibus-gitlab package |
| Azure images | Azure marketplace | Azure image based on the omnibus-gitlab package |
| Kubernetes helm charts | charts/charts.gitlab.io Official Helm Charts | Application definitions for Kubernetes Helm based on the omnibus-gitlab package |
| Redhat Openshift | Openshift template | Template for Openshift Origin based on the omnibus-gitlab package |
| Mesosphere DC/OS package | Universe repository | Package for Mesosphere DC/OS based on omnibus-gitlab package |
| GitLab PCF tile | gitlab.com/gitlab-pivotal | One click installation of GitLab in Pivotal Cloud Foundry based on omnibus-gitlab package |
| GitLab Terraform configuration | gitlab-terraform | Terraform configuration for various cloud providers |
| Omnibus GitLab Builder | GitLab Omnibus Builder | Create environment containing build dependencies for the omnibus-gitlab package |
| Upgrade time metrics | Upgrade time metrics page on GL Pages | Stores the result of calculation of time needed for upgrade between versions in chart form. Backed by Google sheets. Hosted using GL Pages |
Everything that is done in GitLab will end up in the packages that are shipped to users. While that sounds like the last link in the chain, it is one of the most important ones. This means that informing the Distribution team of a change in an early stage is crucial for releasing your feature. While last minute changes are inevitable and can happen, we should strive to avoid them.
We expect every team to reach out to the Distribution team before scheduling a feature in an upcoming release in the following cases:
To sum up the above list:
If you need to do install, update, make, mkdir, mv, cp, chown, chmod, compilation or configuration change in any part of GitLab stack, you need to reach out to the Distribution team for opinion as early as possible.
This will allow us to schedule appropriately the changes (if any) we have to make to the packages.
If a change is reported late in the release cycle or not reported at all, your feature/change might not be shipped within the release.
If you have any doubt whether your change will have an impact on the Distribution team, don't hesitate to ping us in your issue and we'll gladly help.
Every Distribution team member is responsible for creating a training session for the rest of the team. These trainings will be recorded and available to the whole team.
The purpose of team training is to introduce the work done to the rest of your team. It also allows the rest of the team to easily transition into new features and projects and prepare them for maintenance.
Training should be:
Training should not be:
Simply put, the training is a summation of: notes taken in issues during development, programming challenges, high level overview of written documentation. Your team member should be able to take over the maintenance or build on top of your feature with less effort after they have been part of the training.
Note Do not shy away from being technical in your training. You can ask yourself: What would have been useful for me when I started working on this task? What would have helped me be more efficient?
In order to see whether the training is efficient, Distribution lead will rotate team members on projects where training was done. For example, if the feature requires regular releases, the person who gave the training will be considered a tutor. Different team member will follow the training and documentation and will ask the original maintainer for help. The new person responsible is now also responsible for improving the feature. They are now also responsible of training other team members.
Q: Isn't this double work? A: No. The training should be prepared while documenting the task.
Q: Won't this slow me down? A: At the beginning, possibly. However, every hour of the training given will multiple the value of it by the amount of team members.
Q: Isn't it more useful to let the team check out the docs and ask questions? A: In an ideal world, possibly. However, everyone has a lot of tasks assigned and they might not be able to go through the docs until they need to do something. This might be months later and you, as a person who would give the training, might not be able help efficiently anymore.
All work carried out by the Distribution team is public. Some exceptions apply:
If you are unsure whether something needs to remain private, check with the team lead.
Some of the team work is carried out on our development server at dev.gitlab.org. Infrastructure overview document lists the reasons.
Unless your work is related to the security, all other work is carried out in projects on GitLab.com.
General resources available to developers are listed in the Engineering handbook.
In the Distribution team specifically, everyone should have access to the testground project on Google Cloud Platform. If you don't have access, ask the team lead by creating issue in Distribution team issue tracker and label it Access Request.
The process documenting the steps necessary to update the GitLab images available on the various cloud providers is detailed on our Cloud Image Process page.
As part of the team tasks, team has responsibility towards the following nodes:
Every day at 1:30 UTC, a nightly build gets triggered on dev.gitlab.org. The cron trigger times are currently defined at the scheduled pipeline page on dev.gitlab.org.
Every day at 7:20 UTC, the nightly CE packages gets automatically deployed on dev.gitlab.org. Any errors in the install process will be logged in Sentry. Slack notifications will appear in #dev-gitlab. The cron task is currently defined in dev.gitlab.org role.
Announce in #announcements slack channel before and after up/downgrading package on dev.gitlab.org. Something like
Will be manually downgrading package on dev.gitlab.org to <version> as latest nightly shipped some bugs.
Downgrade completed. Also put the package on hold to prevent automatic upgrades.
Will be removing the package hold and manually upgrading package on dev.gitlab.org to <version> (or latest nightly)
Upgrade completed. dev.gitlab.org now runs <version>.
Upgrading
Login to dev and run the following commands
$ sudo apt-get update
$ sudo apt-get install gitlab-ce
and verify if the latest version of the package was installed. Either visit https://dev.gitlab.org/help and confirm the version string there or run the following command
$ apt-cache policy gitlab-ce | grep "Installed"
Downgrading
Sometimes a bug may be introduced in the latest nightly that broke dev.gitlab.org. In such situations, we want to downgrade to a version before the bug was introduced so that dev.gitlab.org is operational again. Most of the times, we will also want to prevent the package from being updated automatically by our cron job until the bug is fixed. To accomplish this, do the following:
$ sudo gitlab-ctl stop sidekiq
$ sudo gitlab-ctl stop unicorn
$ sudo apt-get install gitlab-ce=<version to be installed>
Example:
$ sudo apt-get install gitlab-ce=10.4.0+rnightly.75436.44501791-0
$ sudo gitlab-ctl status
$ sudo apt-mark hold gitlab-ce
$ sudo apt-mark showhold
$ sudo apt-mark unhold gitlab-ce
Requirements:
/etc/gitlab/gitlab.rb, access to the Chef repo. If you do not have access to this repository, make sure you create an issue in Infrastructure issue tracker and label it access request.Teams responsibility is to make sure that the GitLab instance on this server is operational. The omnibus-gitlab package on this server is a stock package with required configuration to keep it operational. Regular omnibus-gitlab commands can be used on this node. If, for some reason, you need to apply a change to /etc/gitlab/gitlab.rb, this change needs to be introduced in the dev-gitlab-org role.
If you do not have access to this repository, but you need to do a hot-patch or configuration testing, the following steps can be performed:
sudo service chef-client stop.gitlab.rb configuration change committed to the Chef server.sudo service chef-client startGitLab CI runner manager is responsible for creating build machines for package builds. This node configuration is managed by cookbook-gitlab-runner. Configuration values are stored in the vault named the same as the node, see example.
Currently, the version of GitLab CI runner is locked. We aim to be close to the current version of runner in order to get the fixes that we need without getting into issues that could cause a failure. These failures could prevent the release from going out so be careful with unnecessary changes on these nodes.
For building official packages we use build-runners.gitlab.org and (soon to be deprecated) omnibus-builder-runners-manager.gitlab.org.
For building packages on GitLab.com as part of trigger packages pipelines, we use a manager machine at build-trigger-runner-manager.gitlab.org.
Both build-runners.gitlab.org and build-trigger-runner-manager.gitlab.org are in GCP project omnibus-build-runners. Each of these managers are spawning machines inside of GCP and are configured with google docker machine driver.
build-runners.gitlab.org is also configured with the scaleway driver, which boots up machines in a Scaleway account for Raspberry Pi (arm platform) builds. This same machine is configured to create package-promotion machines. These machines are used only to upload packages so they are scaled down to save on costs.
omnibus-builder-runners-manager.gitlab.org is currently being used as a backup and is configured with digitalocean docker machine driver and is booting up machines inside of Digital Ocean.
Requirements:
When the version of GitLab CI runner needs to be changed:
bundle exec rake 'edit_role_secrets[<role name for build machine>]'. This command will fetch the secrets from the chef vault and open up your text editor.gitlab-ci-multi-runner section.sudo /root/runner_upgrade.sh to perform the upgrade. This will stop the chef-client service, stop the runner and cleanup the machines, run the chef-client to fetch the new version and finally, start gitlab runner again.When you notice that the builds are pending on our dev.gitlab.org project, it is possible that the number of failed machines is high. Failed machines prevent the runner manager from starting up new machines and this can slow down or even block the release. To resolve this:
sudo su. This is required because docker-machine command will list running machines for currently active userdocker-machine ls. This will print out the list of machines that are either in Running, Error or have an empty state.Error state, you can use /root/machines_operations.sh list-failingError state, run /root/machines_operations.sh remove-failingdocker-machine ls | grep -v 'Running' | awk '{print $1}' | xargs docker-machine rm --force. This line will remove all machines that do not have Running state.At this moment, Distribution team is only the user of packages.gitlab.com. Release packages are served to our users and customers from our CI on dev.gitlab.org.
The duties for this server are yet to be defined with the Production team.
Given that the package server is currently deployed on our own infrastructure, from an omnibus type package, if Production requires help the team should do a best effort to help trough any issues.