Support Engineers need testing environments to learn how to support GitLab and also to replicate customer issues.
This page explains the main choices available to you. Some guidelines:
README.md in the deployment projects, so make sure to give that a read as well.Have fun!
You can always create a group. As usual, public projects get Gold features, but if you need to test paid group features or in a private project, Support has groups for each paid tier.
Typically, Support team members are added as Owner in these groups, while other team members are added as Maintainer or Owner with an expiry date.
If you require access, please ask an existing owner.
The IT Ops team handle provisioning of paid tier test groups on GitLab.com. If you require a group of your own to have a paid tier, please submit an access request. This process will ensure that all entitlements are removed during processes such as staff member offboarding.
If you no longer require a previously provisioned paid tier test group, please submit a new access request for its removal. This ensures that a record of its removal is kept within the same issue tracker as its original creation.
For a test you will need to make an internal issue requesting one.
Please keep in mind you can't generate licenses for customers only for testing purposes.
You can create (ephemeral) testing environments. We recommend using the support-resources project for doing so.
If you need additional flexibility for creating test environments. The GitLab Sandbox Cloud allows for creating a personally-owned GCP project. You can do so using the GCP console, or gcloud command line tool. If you need to replicate any of the Reference Architectures, it's recommended that you use the GitLab Environment Toolkit.
Note: Please remember to shut down resources that you are no longer using. The personally-owned GCP projects aren't yet making use of the frugal resources tool for automating the shut down of resources created in your own project.
support-resources automation (deprecated)You can also use the support-resources project to automatically spin up resources. They will appear in the support-resources GCP project, which all Support Engineers should have access to as part of their baseline entitlements. If you don't have access to this project, please reach out in the #support_operations slack channel for assistance.
support-resources is considered deprecated and we're actively tracking migration to Sandbox Cloud in STM-4037. Resources can still be created here, but if you're looking to create something enduring Sandbox Cloud should be preferred.
Some advantages of using the support-resources automation project over Sandbox Cloud are:
terraform, ansible and general automation guidelines (could include here gcloud, bash and chef). This is becoming more and more important as we are seeing the adoption of GET.There's also a few disadvantages to using the project:
Support-Resources automation project is good at preventing resources conflicts, those can still happen as everybody has access to the pool of resources (so accidents can and have happened). In a personal GCP project one can, among other things, restrict access to their resources.Support-Resources project a lot of the intricacies of using GCP are performed beneath the covers. Having your own GCP projects will expose you to the complexity of setting things up manually or automating that yourself.You can use the support-resources project to manually create resources in a GCP testing environment alongside the resources created by our automation tools. As with the GitLab Sandbox Cloud for GCP - you can manage these manually created resources using the GCP console, or gcloud command line tool.
Warning: You may also have access to the gitlab-internal and gitlab-support GCP projects. It's strongly recommended that you make use of the support-resources project or the GitLab Sandbox Cloud, instead of creating new resources in these projects.
Note: Please remember to shut down any manually created that you are no longer using.
Please use the support-resources GCP project or your GitLab Sandbox Cloud GCP project to create a GCP Kubernetes (GKE) cluster. If you are using the support-resources GCP project you can create a GKE cluster manually from the console or you can use the Support-resources automation project to streamline your GKE cluster creation and there's even an option to create a GKE cluster that already has GitLab installed through helm chart.
Note: If you are using GKE to test GitLab Runners, it is recomended to use GitLab Sandbox Cloud. GitLab Runners required the use of RBAC roles in GCP. The support-resources GCP project does not allow the user enough permision to create the required roles.
All of the remaining options can be left as their default settings unless you have a need to add customization to your cluster. Of note, the Maximum Pods per Node option directly correlates with the CIDR assignment of your node(s).
Connecting to, and configuring, your cluster can be done locally using gcloud and kubectl. Or you can use the Google Cloud Shell. Clicking Connect in GCP will provide the command to run locally for you to copy and paste, or let you open Cloud Shell immediately. Using docker images is also an option for a quick deployment of all tools locally.
Please use Sandbox Cloud to create credentials for AWS.
The following guide is designed to help you quickly deploy a Kubenertes environment using EKS and Helm for testing purposes on this environment.
Pre-requisites:
Install the following programs on your local computer:
For OS X, you can install all of the programs above by running:
brew install certbot helm kubectl
When they key has been created successfully, you'll get an Access key ID and a Secret key.
Go to your terminal to configure the AWS CLI to use the IAM Access Key:
aws configurejson, yaml and text.)A bootstrap script has been created to setup an easier and automated cluster creation using Cloud Formation.
Clone this project locally and navigate to gitlab/scripts and execute ./eks_bootstrap_script up. This script will use a CloudFormation stack to deploy all the required resources for your cluster so it will take a while to complete. After successful completion you will see something similar this output:
2022-07-18 16:47:40 [ℹ] node "ip-192-168-63-211.us-east-2.compute.internal" is ready
2022-07-18 16:47:41 [ℹ] kubectl command should work with "/Users/cats/.kube/config", try 'kubectl get nodes'
2022-07-18 16:47:41 [✔] EKS cluster "gitlab-cluster" in "us-east-2" region is ready
Once your cluster is ready create a certificate for your domain using Certbot by running:
sudo certbot --server https://acme-v02.api.letsencrypt.org/directory --manual --preferred-challenges dns-01 certonly -d "*.<DOMAIN NAME>"
Certbot will create a TXT DNS record that you need to add on your DNS Registrar so that your domain will have a valid HTTPS certificate to be used by your GitLab instance.
After your record had been added to your registrar create a secret for your cluster to use the certificate by running:
sudo kubectl create secret tls tls-star-<SECRET NAME> --cert=/etc/letsencrypt/live/<DOMAIN NAME>/fullchain.pem --key=/etc/letsencrypt/live/<DOMAIN NAME>/privkey.pem
(Optional) If you also want to have GitLab Pages in your cluster then you will need to create another certificate for your domain and another secret for your cluster
sudo certbot --server https://acme-v02.api.letsencrypt.org/directory --manual --preferred-challenges dns-01 certonly -d "*.pages.<DOMAIN NAME>"
sudo kubectl create secret tls tls-star-pages-<SECRET NAME> --cert=/etc/letsencrypt/live/pages.<DOMAIN NAME>/fullchain.pem --key=/etc/letsencrypt/live/pages.<DOMAIN NAME>/privkey.pem
helm repo add gitlab https://charts.gitlab.io/
helm repo update
values.yml file with the desired values you wish to deploy your instance. (A sample file can be found here)helm install gitlab gitlab/gitlab -f <values.yml>
kubectl get ingress/gitlab-webservice-default -ojsonpath='{.status.loadBalancer.ingress[0].hostname}'
gitlab pointing to your loadbalancer hostname to access your instance on HTTPS.*.pages pointing to the same loadbalancer hostname.kubectl get secret gitlab-gitlab-initial-root-password -ojsonpath='{.data.password}' | base64 --decode ; echo
:tada: Your GitLab instance is up and running!
Once these steps are completed you can visit your GitLab instance using gitlab.<DOMAIN NAME>.
./eks_bootstrap_script down to destroy all resources used../eks_bootstrap_script up.Note: You might need to wait for your TTL to reset after you updated your loadbalancer in your registrar before accessing the instance.
For Group SAML/SCIM (GitLab.com) testing, shared account credentials can be located within the 1password entry Azure Active Directory Sandbox (SAML Testing). This level of access should be sufficient for the majority of test cases.
If you need access to the shared infrastructure team's Azure account for testing AKS and similar features, please create an access request and tag your manager.
For features not available in either instance, please create your own trial for Azure.
If you need an Okta sandbox environment to test SAML and SCIM, please go to https://developer.okta.com/signup/ and enter your credentials for a free developer instance where you can perform all your tests. This is a full featured environment so you should be able to add and remove applications, and perform all tests as if you were in a regular production instance.
For testing LDAP integrations with a self-managed GitLab instance, you may consider any of these options:
If you wish to test resources using a real domain name (instead of an IP address, e.g. for testing TLS certificates), you can use a subdomain of gitlab.support. You can configure this in GCP in the gitlab-support project here.
Test instances are, by default, publicly accessible on the Internet. Often, we need to test specific versions or configurations that may be vulnerable to remote compromise. It is your responsibility to secure your test instances to prevent them from being compromised and used to further attack our cloud environment.
A highly effective way to secure your cloud instances is to apply the concept of IP filtering for each test instance you create whether its a GitLab instance or otherwise. For the majority of cases, this means source IP filtering from one or more CIDR block ranges ensuring that only certain IPs and integrations can interact with the GitLab instance, therefore reducing the attack surface of the GitLab organization as a whole.
If you don't know your current IP address to use for source IP filtering, you can utilize services like whatsmyipaddress.com or ipinfo.io to retrieve it. Though configuring IP filtering is currently a manual process for each cloud service, discussions are underway at the time of writing regarding how best to automate this practice to reduce complexity and manual processes needed in some cases. The steps to implement IP filtering will differ per cloud environment. Below you can find a detailed guide maintained by the support engineering team.
In addition, you can find official, platform-specific documentation of features involved in implementing IP filtering:
To be in line with GitLab's encryption policy, TLS should also be implemented on public-facing testing resources. Implementing TLS on any test instance that includes a login page can be done with a self-signed certificate if desired. Self-signed certificates are free, suitable for testing environments, and encrypt ingress and egress traffic with the same ciphers as paid certificates. The down-side is that self-signed certificates are not trusted by any browser or operating system and will therefore warn users of the risks when accessing a site that utilizes a self-signed (untrusted) certificate. If external parties will be accessing your instance that should rely on your TLS implementation, it's best to include a signed certificate from a legitimate certificate authority. For instructions on using self-signed certificates on your test instances, please review the following documentation:
For GitLab instances specifically, it's recommended that LetsEncrypt is manually enabled for Omnibus installs. First, you'll need a domain assigned to your external IP regardless of your cloud platform. Enabling LetsEncrypt for Omnibus installs is well documented at the link above.
Running a Support test instance with a known vulnerability can be a security issue. When running a test instance, you should patch against known vulnerabilities if possible.
For example, GitLab versions from 11.9 to 13.10.2 are vulnerable to CVE-2021-22205.
If you're running GitLab version 12.6 to 13.10.2, patch against this vulnerability by running the following commands for each instance:
sudo su
cd ~
curl -JLO https://gitlab.com/gitlab-org/build/CNG/-/raw/master/gitlab-ruby/patches/allow-only-tiff-jpeg-exif-strip.patch
cd /opt/gitlab/embedded/lib/exiftool-perl
patch -p2 < ~/allow-only-tiff-jpeg-exif-strip.patch
For GitLab versions 11.9.0 to 12.5.x, you can "patch" the vulnerability by replacing the exiftool binary with the following commands:
sudo rm -f /opt/gitlab/embedded/bin/exiftool
sudo printf '#!/bin/bash \n\ncat -' > /opt/gitlab/embedded/bin/exiftool
sudo chmod a+x /opt/gitlab/embedded/bin/exiftool
If you'd like to use Docker Desktop for Mac a subscription is required for business use. Please review the Docker Desktop handbook page to find more information on how to obtain a license as well as a list of recommended alternatives.
In the mean time, consider using a Cloud or local VM with Linux Engine for testing Docker environments.
Note that on Macs with M1 / Apple Silicon, running GitLab in Docker is not working properly for now. Check out UTM below as an alternative for a local setup on your M1 Mac.
Since Docker Toolbox has been deprecated, Docker Machine has to be downloaded and installed manually. Use the following commands to install or upgrade Docker Machine separately:
$ curl -L https://github.com/docker/machine/releases/download/v0.16.2/docker-machine-`uname -s`-`uname -m` >/usr/local/bin/docker-machine && \
chmod +x /usr/local/bin/docker-machine
This guide involves configuring and setting up VMWare and Docker locally and assumes you're using macOS.
UTM is an open source QEMU-based virtualisation software for MacOS. It currently provides support for Apple silicon hardware, so may be useful for engineers on Apple M1 devices to deploy local virtual machines.
This guide involves configuring and setting up VirtualBox and Docker locally and assumes you're using macOS. VirtualBox does not currently support Apple M1 CPUs.
Oracle VM VirtualBox is a free and open-source hosted hypervisor for x86 virtualization.
Note The following list of commands can be saved as bash script for quickly spinning up new instances
Vagrant is a tool for building and managing virtual machine environments in a single workflow
Vagrant encapsulates the local VM apps VMWare and Virtual along with libvirt.
To install Vagrant, go to tutorials/vagrant/getting-started-install
Once installed, support/toolbox has two projects which you can explore for local GitLab and tools setup.
Support toolkit to help manage GitLab inventory and additional services via docker containers.
Provide a common provisioning and directory structure for various support setups including GitLab with connected GitLab Runners.
export SSH_PORT=2222
export HTTP_PORT=8888
export VERSION=11.9.9-ee.0
export ENV_NAME=gitlab-test-env
export CONTAINER_NAME=gitlab-test-11.9
This command will create a new VirtualBox virtual machine called gitlab-test-env that will act as a docker host.
-1)4GBgitlab-test-envvirtualboxdocker-machine create \
--virtualbox-cpu-count -1 \
--virtualbox-memory 4096 \
--virtualbox-disk-size 30000 \
--driver virtualbox $ENV_NAME
In this example we'll create a GitLab EE 11.9.9 instance.
First connect the docker client to the docker host you created previously.
eval "$(docker-machine env gitlab-test-env)"
You can add this to your ~/.bash_profile file to ensure the docker client uses the gitlab-test-env docker host. You can use echo to do so:
echo 'eval "$(docker-machine env gitlab-test-env)"' >> ~/.bash_profile
Optionally replace the ee in the URL with ce
wget -q https://registry.hub.docker.com/v1/repositories/gitlab/gitlab-ee/tags -O - | sed -e 's/[][]//g' -e 's/"//g' -e 's/ //g' | tr '}' '\n' | awk -F: '{print $3}'
88882222
gitlab_shell_ssh_port using --env GITLAB_OMNIBUS_CONFIGgitlab-test-11.911.9.9-ee.0export IP=$(docker-machine ip $ENV_NAME)
docker run --detach \
--env GITLAB_OMNIBUS_CONFIG="external_url 'http://$IP:$HTTP_PORT'; gitlab_rails['gitlab_shell_ssh_port'] = $SSH_PORT;" \
--hostname $IP \
-p $HTTP_PORT:$HTTP_PORT -p $SSH_PORT:22 \
--name $CONTAINER_NAME \
gitlab/gitlab-ee:$VERSION
echo $IP
# example output: 192.168.151.134
Note: The container might take a few seconds to spin up and become accessible via the browser.
docker exec -it $CONTAINER_NAME /bin/bash
# example commands
root@192:/# vi /etc/gitlab/gitlab.rb
root@192:/# gitlab-ctl reconfigure
For example, to set the gitlab_shell_ssh port on a container named gitlab-ee to port 2222
docker exec -it gitlab-ee \
sed -i "s/.*gitlab_shell_ssh_port.*/gitlab_rails['gitlab_shell_ssh_port'] = 2222/g" /etc/gitlab/gitlab.rb
docker exec -it gitlab-ee gitlab-ctl reconfigure
It may come to pass that you require a Windows environment to test a Windows Runner or even the Kubernetes Executor in a Mixed Cluster.
The options are the same as above: