This page lists a number of tips and tricks we have found useful in day to day Quality Engineering related tasks.
While working on the GitLab-QA codebase, it is sometimes helpful to run the GitLab-QA pipeline
against a specific release of the GitLab project. This could be
due reasons such as that particular GitLab release containing specific code needed for validating the changes made
in GitLab-QA. To run a GitLab-QA pipeline against
a specific GitLab release, we need to know the GitLab release version created and tagged by the omnibus pipeline.
This can be found by either observing the RELEASE variable in any of the package-and-test test jobs or
in the last output line of the Trigger:gitlab-docker job triggered by the package-and-test job. Here is an example of what the RELEASE string
looks like:
registry.gitlab.com/gitlab-org/omnibus-gitlab/gitlab-ee:41b42271ff37bf79066ef3089432ee28cfd81d8c
Copy this string and create a new GitLab-QA pipeline
with a RELEASE variable and use the copied string as its value. Create another variable called QA_IMAGE and set it to the value
that can be found in the package-and-test upstream job. Here is an example of what the QA_IMAGE string looks like:
registry.gitlab.com/gitlab-org/gitlab/gitlab-ee-qa:qa-shl-use-unique-group-for-access-termination-specs
Note that the string is the same as RELEASE except for the -qa suffix on the image name and the tag which is the branch name on the GitLab project.
Now run the pipeline against the branch that has your changes.
It's also possible to trigger a manual GitLab-QA pipeline against a specific GitLab environment using the RELEASE and QA_IMAGE variable from the package-and-test job of GitLab's Merge Request.
For example, here is the link to run a manual GitLab QA pipeline against Staging.
registry.gitlab.com is used, you will also need to include the GITLAB_QA_CONTAINER_REGISTRY_ACCESS_TOKEN variable with the value set to the production gitlab-qa user's access token to avoid authentication errors.It is often needed to test the impact of changes in the GitLab-QA codebase on
Nightly, Staging,
Pre-Prod, Canary
or Production pipelines.
This can be achieved by manually triggering a pipeline in any of these projects and setting the QA_BRANCH variable to the branch name you are working on in the GitLab-QA project.
As a result, the pipeline will checkout the specified branch and build the gitlab-qa gem instead of using the latest published gem.
You can checkout a test branch and edit the Gemfile to change the gitlab-qa line to install via the Gitlab-QA branch.
For example in the qa/gemfile:
gem 'gitlab-qa', git: 'https://gitlab.com/gitlab-org/gitlab-qa.git', branch: '<Gitlab-QA-branch>'
Make sure to also bundle install and commit the Gemfile.lock as well.
Doing so successfully will allow the gitlab-qa gem to be built from a custom branch.
To find out the version, revision, branch and package deployed in gitlab.com, staging and canary environments, run this in the #chat-ops-test Slack channel:
/chatops run auto_deploy status
You will need access to the https://ops.gitlab.net/gitlab-com/chatops project to run /chatops commands.
Ask to be added to this project in the #development Slack channel.
If you have a revision SHA that is deployed on an environment, you can find out if a change has made it to that environment.
For example, if the revision SHA deployed on an environment is c46489109e4 and you want to find out if a change in
restrict_by_ip_address_spec.rb has made it there, you can use:
git show c46489109e4:qa/qa/specs/features/ee/browser_ui/1_manage/group/restrict_by_ip_address_spec.rb
You can determine the revision SHA deployed on a GitLab instance by either navigating to www.example.com/help,
by calling the http://www.example.com/api/v4/version API or by running /chatops run auto_deploy status in a Slack
channel such as #chat-ops-test.
You can also determine if your commit has been deployed on a GitLab environment using ChatOps.
For example, if your commit ref is 347e530c5b3dec60c0ce2870bc79ca4c8273604d you can run this command in a Slack
channel such as #chat-ops-test:
/chatops run auto_deploy status 347e530c5b3dec60c0ce2870bc79ca4c8273604d
The commit SHA for a nightly pipeline can be determined in the following ways:
/api/v4/version APIRun the nightly docker image
docker run \
--hostname localhost \
--publish 443:443 --publish 80:80 --publish 22:22 \
gitlab/gitlab-ee:nightly
The commit SHA can be determined by visiting the http://localhost/help page after sign-in
or by calling the /api/v4/version API where it is displayed as a value of the revision attribute.
Nightly images are created by scheduled pipelines here: https://dev.gitlab.org/gitlab/omnibus-gitlab/pipeline_schedules
You can look at the last pipeline by clicking the pipeline number for CE nightly or EE nightly under the "Last pipeline" column.
In the pipeline view click a job under the "Gitlab_com:package" column. The SHAs for GitLab Components
are listed towards the end of the logs. The GitLab commit SHA is displayed as a value of gitlab-rails.
The Ruby VS Code extension adds a few Ruby-related capabilities to VS Code, including the ability to debug Ruby code.
After you install the extension you can use VS Code to debug end-to-end specs running against your local GDK. You will need to add a Run configuration to launch.json. For example, the following launch.json will add a configuration named Debug Test::Instance::All current file to the list in the Run view of your Sidebar. Then, with a spec file open in the editor you can start debugging (F5) and VS Code will run the tests in the spec file.
{
"version": "0.2.0",
"configurations": [
{
"name": "Debug Test::Instance::All current file",
"type": "Ruby",
"request": "launch",
"useBundler": true,
"pathToBundler": "<path_to_bundler>",
"cwd": "${workspaceRoot}/qa",
"program": "${workspaceRoot}/qa/bin/qa",
"env": {
"CHROME_HEADLESS": "false",
"QA_DEBUG": "true"
},
"args": [
"Test::Instance::All",
"http://localhost:3000",
"--",
"${file}"
]
}
]
}
You can include multiple configurations, and any environment variables or command line options can be included. For example, here's one that will debug smoke tests while running them on Staging:
{
"name": "Debug Staging Smoke tests",
"type": "Ruby",
"request": "launch",
"useBundler": true,
"pathToBundler": "<path_to_bundler>",
"cwd": "${workspaceRoot}/qa",
"program": "${workspaceRoot}/qa/bin/qa",
"env": {
"CHROME_HEADLESS": "false",
"QA_DEBUG": "true",
"GITLAB_USERNAME": "gitlab-qa",
"GITLAB_PASSWORD": "from 1Password",
"GITLAB_QA_ACCESS_TOKEN": "from 1Password"
},
"args": [
"Test::Instance::All",
"https://staging.gitlab.com",
"--",
"--tag", "smoke"
]
}
Sometimes, it's useful to deploy auto-scaled runners to try out and compare different machine types.
To do so, follow these steps:
gitlab-qa-resources GCP project.
gitlab-qa-resources GCP project.
scp ~/Downloads/gitlab-qa-resources-abc123.json <your-username>@VM-IP:/home/<your-username>).gitlab-qa-resources GCP project.gitlab-runner.docker+machine as the runner executor.sudo): sudo mv home/<your-username>/gitlab-qa-resources-abc123.json /etc/gitlab-runner/service-account.json[runners.machine] documentation): concurrent = 500
check_interval = 0
[session_server]
session_timeout = 1800
[[runners]]
name = "n2d-highcpu-4 Relevant description of the runner manager"
url = "https://gitlab.com/"
token = "[REDACTED]"
executor = "docker+machine"
limit = 500
pre_clone_script = "eval \"$CI_PRE_CLONE_SCRIPT\""
request_concurrency = 500
environment = [
"DOCKER_TLS_CERTDIR=",
"DOCKER_DRIVER=overlay2",
"FF_USE_DIRECT_DOWNLOAD=true",
"FF_GITLAB_REGISTRY_HELPER_IMAGE=true"
]
[runners.custom_build_dir]
enabled = true
[runners.cache]
Type = "gcs"
Shared = true
[runners.cache.gcs]
BucketName = "BUCKET-NAME-FOR-THE-SHARED-CACHE"
CredentialsFile = "/etc/gitlab-runner/service-account.json"
[runners.docker]
tls_verify = false
image = "ruby:2.7"
privileged = true
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
shm_size = 0
volumes = [
"/cache",
"/certs/client"
]
[runners.machine]
IdleCount = 0
IdleTime = 600
MachineDriver = "google"
MachineName = "rymai-n2d-hc-4-%s"
MachineOptions = [
# Additional machine options can be added using the Google Compute Engine driver.
# If you experience problems with an unreachable host (ex. "Waiting for SSH"),
# you should remove optional parameters to help with debugging.
# https://docs.docker.com/machine/drivers/gce/
"google-project=gitlab-qa-resources",
"google-zone=us-central1-a", # e.g. 'us-central1-a', full list in https://cloud.google.com/compute/docs/regions-zones/
"google-machine-type=n2d-highcpu-4", # e.g. 'n1-standard-8'
"google-disk-size=50",
"google-disk-type=pd-ssd",
"google-label=gl_resource_type:ci_ephemeral",
"google-username=cos",
"google-operation-backoff-initial-interval=2",
"google-use-internal-ip",
"engine-registry-mirror=https://mirror.gcr.io",
]
OffPeakTimezone = ""
OffPeakIdleCount = 0
OffPeakIdleTime = 0
MaxBuilds = 20
[[runners.machine.autoscaling]]
Periods = ["* * 8-18 * * mon-fri *"] # During the weekends
IdleCount = 0
IdleTime = 600
Timezone = "UTC"
The Quality Toolbox contains several scripts that can be useful when working with GitLab end-to-end tests, such as one to generate a report of flaky tests, or one to report job success rates.
The qa/tools directory contains rake tasks that perform automated tasks on a schedule (such as deleting subgroups after a test run), or that can be run as needed (such as revoking personal access tokens).
This script deletes SSH keys for a specific user. It can be executed via the delete_test_ssh_keys rake task in the qa directory.
The rake task accepts three arguments that can be used to limit the keys that are deleted, and to perform a dry run.
title_portion, limits keys to be deleted to those that include the string provided.delete_before, limits keys to be deleted to those that were created before the given date.dry_run, determines if the command will be executed as a dry run, summarizing the keys to be deleted. Set to true to execute as a dry run.Two environment variables are also required:
GITLAB_ADDRESS is the address of the target GitLab instance.GITLAB_QA_ACCESS_TOKEN should be a personal access token with API access and should belong to the user whose keys will be deleted.For example, the following command will delete all SSH keys with a title that includes E2E test key: and that were created before 2020-08-02 on https://staging.gitlab.com for the user with the provided personal access token:
GITLAB_QA_ACCESS_TOKEN=secret GITLAB_ADDRESS=https://staging.gitlab.com bundle exec rake "delete_test_ssh_keys[E2E test key:, 2020-08-02]"
There is a rake task to set the default password (from Runtime::User.default_password) if it's not set already, and it creates a personal access token.
This is useful when testing on a fresh GitLab instance (e.g., an omnibus-gitlab docker image) and you don't want to have to log in and create an access token manually.
Usage example (run from the gitlab/qa directory):
$ bundle exec rake 'initialize_gitlab_auth[http://gitlab.test]'
Signing in and creating the default password for the root user if it's not set already...
Creating an API scoped access token for the root user...
Token: s8xbMN3qMjyUyQATDWgp