GitLab Dedicated, from a support perspective, works as a combination of SaaS and Self-Managed. Customers have full Admin access to the instance, but no access to the infrastructure, nor to the backend configurations. This workflow captures the differences, and details of providing support for GitLab Dedicated.
If you'd like to work on GitLab Dedicated tickets, consider creating an issue using the template in Support Training, and read the overview.
The GitLab Dedicated team does not have administrative access to the Admin Area in the GitLab application on Dedicated instances. Any support requests that require a GitLab instance administrator to make a change in the Admin Area, for example resetting 2FA, has to be performed by the appropriate team within the customer organization.
Support can access tenant logs through OpenSearch.
OpenSearch can be used similar to Kibana.
Credentials needed for examining logs are stored in the GitLab Dedicated - Support vault. Each customer is noted by a customer number in the vault, so you must refer to the <tenant name> to identify the proper credentials to use for a customer. This is used as part of the accessible URL, such as: opensearch.<tenant name>.gitlab-dedicated.com.
To access the logs for a specific tenant find the credentials stored in the GitLab Dedicated - Support Vault, and access the corresponding tenant URL listed there.
Once in the tenant's OpenSearch site:
gitlab-* is selected.NOTE: Logs are rotated weekly. Copy and paste relevant logs into the ticket and/or issues. Screenshots of frequently occurring errors over time are also helpful.
Since GitLab Dedicated uses Cloud Native Hybrid reference architecture, searching logs on OpenSearch is a bit different from Kibana.
General fields:
host: The GitLab host of the log. It can be <tenant name>-gitaly-* or <tenant name>-consul-2, etc.referrer: holds the project path. https://tenant.gitlab-dedicated.com/example-group/test123message: is the message that would be seen in the logs of a self-managed instance. xxx.xxx.xxx.xxx - - [08/Jul/2020:13:24:43 +0000] "GET /assets/webpack/commons-pages.projects.show-pages.projects.tree.show.21909065.chunk.js HTTP/1.1" 200 9316 "https://tenant.gitlab-dedicated.com/example-group/test123" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.5005.63 Safari/537.36" 1343 0.001 [default-gitlab-webservice-default-8181] [] xxx.xxx.xxx.xxx:8181 9309 0.000 200 fe130eac78314cwf352g3762397572cbkubernetes.labels.app: used to filter Kubernetes pods. nginx-ingress, webservice, etc.Gitaly related fields:
grpc.request.glProjectPath: The actual GitLab path project path.grpc.request.repoPath: Project hash id path.SAML related fields:
action:samlpath: /users/auth/saml/callbackcontroller: OmniauthCallbacksControllerlocation: https://tenant.gitlab-dedicated.com/Credentials are stored in the GitLab Dedicated - Support Vault. Just as with OpenSearch, the passwords are refered to by customer number. Use the website field to find the customer.
When logging in for the first time, dashboards are not immediately visible and you will be greeted by a Grafana welcome screen. To find the dashboards:
General. You can expand this folder to see the dashboards.The General / Triage dashboard is most useful for an emergency as it has the pods all laid out in a single view. By default it has 6 hours of data. It is helpful for finding blips & dips. Use this data to correlate to other dashboards.
Remember that Grafana is used for visualizing issues and spotting problems. It won't tell us directly what is wrong. You must correlate to the logs to find the exact problem.
As an example, the Triage dashboard may show that webservice errors are increasing. Use this to correlate an approximate time and filter out the other logs by the kubernetes.labels.app to find the error for just webservice.
GitLab Dedicated tenants are defined in the
Switchboard repository's tenant_models directory.
json file.json
file's commit history and search for commits that mention gitlab.GitLab Dedicated uses the Cloud Native Hybrid reference architecture. Instance implementation and changes are done via the instrumentor project
When any changes to the tenant instance are required, raise an issue in the GitLab Dedicated issue tracker using a Support Request template.
If it's an emergency, escalate the emergency and contact GitLab Dedicated infrastructure team on Slack, using channel #g_dedicated-team.
In cases where Customer Support needs to interact with GitLab Dedicated engineers to gather information or similarly debug a problem at tenant's request (when Grafana or OpenSearch does not suffice), raise an issue in the GitLab Dedicated issue tracker using a Support Request template.
Emergencies from GitLab Dedicated will come through the Customer Emergencies On-call Rotation as with other emergency types.
The GitLab Dedicated Infrastructure team has a 24/7 PagerDuty rotation: GitLab Dedicated Platform Escalation. To manually create a PD Incident use the Dedicated Platform Service or use the Slack command /pd trigger and choose "Dedicated Platform Service" as the Impacted Service to escalate an emergency to an SRE after initial triage and analysis.
Customers can add a custom identifier, such as the ticket ID, to the user-agent field when testing. This makes it easier to filter logs related to the test.
For example:
curl -k -vvv -A"GitLabSupport012345" "https://tenant.gitlab-dedicated.com/users/sign_in"