The Support Operations team maintains the day-to-day operations and software systems used by GitLab’s global Technical Support team.
At GitLab, CSAT is a measure of how satisfied our customers' are with their interaction with the GitLab Support team. This is based on survey responses from customers after each ticket is solved by the Support team using a Good or Bad rating.
The target is 95% customer satisfaction. The definition of the KPI is:
Satisfied [total good scores] / Total Survey Responses [good scores + bad scores]
This calculation excludes cases where a survey was not offered or where it was offered but no score was provided.
The survey is sent out 24 hours after a ticket is solved. The tag csat_survey_sent is added to a ticket as soon as a survey is sent out.
There are a few reasons why a customer wouldn’t receive a CSAT Survey after a ticket has been solved:
close_by_merge).
Starting July 9th, 2019, in order to learn about the reasons behind a bad satisfaction rating, we ask any customer who gives a negative survey response to select a reason for their dissatisfaction. This feedback is shared with the Product team and the Support Managers, depending on which reason is chosen. The dropdown options are:
All negative and positive feedback with comment is exported to an internal repository. Once the feedback has been submitted, an Issue is created and the Support Engineering Managers are tagged when a negative rating has been submitted. It is the responsibility of the Support Engineering Managers to investigate the feedback and ensure appropriate actions are taken to either resolve the root cause of the negative experience or reduce the likelihood of it recurring.
We have the option of allowing all of the end-users in an organization to see each other's tickets. This is referred to as a shared organization in Zendesk.
We can grant the end-users permissions to see the tickets at the organization level and the personal level (personal level overrides org level). This means that at the personal level, we can allow selected individuals in an org to access all the tickets created by members of their org or we can allow everyone in the company to see everyone's tickets.
Shared organizations will only be set up if we receive a request directly from the customer through a Support Ticket. Once the ticket is created, any member of the Customer Success or Support teams can create a request in the Support Ops project using the Shared Organization Request template, and any member of the Support Team can update the organization settings once the issue is created.
There are a few different options available for our customers:
We have enabled Zendesk to send all ticket information to Salesforce using a target. Currently, we create a Salesforce Zendesk Ticket record whenever a ticket is created and when it is solved by one of our agents or an automation. Once the information is received in Salesforce, a notification is sent from SFDC to the Technical Account Managers so they are aware of the issues their accounts are experiencing.
To troubleshoot any issues with this functionality, we need to make sure that:
Use auto-matching criteria for searching Salesforce recordsZendesk Support Ticket and Zendesk Support Ticket Comment is visible to the System AdministratorWhen working on the Salesforce side of this configuration, make sure to always communicate with the Sales Systems and Operations team about any inconsistencies you find and request their help to troubleshoot any issues.
All organizations in Zendesk are created through our Salesforce Integration. Currently, only records with Account Type equal to Customer are synced from Salesforce to Zendesk.
Accounts are synced with Organizations as long as:
Account ID in SFDC matches the one Salesforce ID in ZendeskZendesk Support Organization ID in SFDC matches the Zendesk Organization ID in Zendesk.Zendesk Support Organization name in SFDC matches the Zendesk Organization name.We have configured custom field mappings:
A script was created by the Support Team in order to do routine checks of data integrity between Salesforce and Zendesk. This script runs daily and it checks for:
missingzendeskorg_dotcom.rb) and Self-Managed Customers (missingzendeskorg_selfmanaged.rb)nosupportlevel)repeatrecordsinsalesforce)supportlevelmismatch)nosalesforceid
)org_name_mismatch)We run a Salesforce to Zendesk bulk sync twice or three times a week because we have a few issues with the ongoing sync not working as expected. The main issue with the ongoing sync is that all our new organizations in Zendesk are created with a Starter GitLab plan (even if they are Priority customers) and we need to use the bulk sync to update all new accounts to their correct GitLab Plan. To run a bulk sync:
Zendesk for Salesforce appZendesk Support Bulk SyncSalesforce to ZendeskAccount and OrganizationFull syncNoneStart Sync.The process may take around 2 hours, so you can leave it running in the background.
PagerDuty is how we dispatch emergencies and handle scheduling for different roles.
A service in PagerDuty usually represents a component a team monitors. For Support's purposes, however, it's just
something you'd go on call for. Services interact with /chatops oncall commands to display who is currently on-call
in Slack.
The Customer Support Service is the service we use to track who is on-call for Emergency Support.
For more information, read more about how being on-call works.
This service is driven by Support Engineers participating in the Customer Emergency Rotation
An incident can be generated by:
When an incident occurs, it will alert in #support_self-managed on Slack.
The Customer Supoprt - US Federal Service is the service we use to track who is on-call for Emergency Support for US Federal Customers.
This service is driven by US Citizens participating in the US Federal Customer Emergency Rotation
An incident can be generated by:
When an incident occurs, it will not alert in Slack.
The FRT Hawk Service is what we use to track who is responsible for playing the role of FRT Hawk.
This service is driven by Support Engineers participating in the Customer Support - FRT Hawk Rotation.
This Service has no integrations and is used only for scheduling.
The SLA Hawk Service is what we use to track who is responsible for playing the role of SLA Hawk.
This service is driven by Support Engineers participating in the Customer Support - SLA Hawk Rotation.
This Service has no integrations and is used only for scheduling.
The Support Managers Service is what we use to track which Support Manager is on-call.
For more information, read about what a Support Manager On-call does
This service is driven by Support Manager participating in the Support Managers Rotation.
An incident can be generated by:
/pd-support-manager in SlackWhen an incident occurs, it will alert in #support-managers in Slack.
An escalation policy is built out of PagerDuty Schedules with rules that define when things should be moved to the next layer.
The Customer Emergency Rotation defines who should be paged in the event of a customer emergency.
Immediately after an incident is triggered, notify:
- Customer Emergencies - AMER
- Customer Emergencies - APAC
- Customer Emergencies - EMEA
Escalate after 10 minutes:
- Support Manager - AMER
- Support Manager - APAC
- Support Manager - EMEA
Escalate after 5 minutes:
- Customer Emergencies EMEA Escalation
- Drew, Arihant, Harish, Cindy, Athar, Aric, Davin, Adam
Repeat 5 times if no one acknowledges incidents
The US Federal Customer Emergency Rotation defines who should be paged in the event of a US Federal customer emergency.
Immediately after an incident is triggered, notify:
- US Federal On-Call
Escalate after 15 minutes:
- Support Manager - AMER
Escalate after 20 minutes:
- Lyle, Lee, Jason
The Customer Support - FRT Hawk Escalation Policy defines who is playing the role of FRT Hawk.
It shouldn't ever page, as it's used only to facilitate scheduling.
Immediately after an incident is triggered, notify:
- FRT Hawk - AMER Easterly
- FRT Hawk - AMER Westerly
- Customer Emergencies - APAC
- Customer Emergencies - EMEA
The Customer Support - SLA Hawk Escalation Policy defines who is playing the role of SLA Hawk.
It shouldn't ever page, as it's used only to facilitate scheduling.
Immediately after an incident is triggered, notify:
- SLA Hawk - AMER Easterly
- SLA Hawk - AMER Westerly
The Support Managers Escalation Policy defines which Support Manager is on call.
Immediately after an incident is triggered, notify:
- Support Manager - AMER
- Support Manager - EMEA
- Support Manager - APAC
PagerDuty schedules drive the actual rotations of individuals and track overrides.
For ease, we define the hours of on-call duty per region as follows:
Hand-off: Monday at 07:00 UTC.
At writing (2019-10-04) there are no further subdivisions per region.
The following schedules are used:
Customer Emergencies
US Federal Customer Emergencies
FRT Hawks
SLA Hawks
Support Managers
You can subscribe to a WebCal feed, suitable for viewing in Google Calendar.
From any of the above links you can subscribe to the whole schedule by clicking Schedule Info and then WebCal feed. Alternatively you can pull just your schedule from general Schedules page by clicking Export then Just my calendar in the WebCal Feed section.
If you want just one calendar for all of your on-call, you can grab a WebCal feed by clicking on your profile picture, going to My Profile and then On-Call Shift, then clicking the Export button to reveal the link.
The premium breach notifications is a webhook that is triggered by a ZenDesk automation titled Premium important ticket slack notification. The trigger looks for any
high premium tagged tickets that have less than 2 hours left on the breach clock.
The automation runs at the top of every hour
and due to this limitation, it does not run at the exact hour the ticket is less than 2 hour from breaching.
Once the ticket is updated, a trigger sends a webhook to slack which is configured on the slack side.
The #zd-self-managed-feed channel is updated by a webhook that's triggered whenever a Self-Managed ticket is created or updated. The Zendesk trigger is called "Slack Support Live Feed (With Organization names)"
The #zd-self-managed-feed channel does not display tickets addressed to GitLab.com or
GitHost.io.
The #zd-gitlab-com-feed channel is updated by a webhook that's triggered whenever a GitLab.com ticket is created or updated. The Zendesk trigger is called "Slack Services Live Feed". This channel does not display EE tickets.
The Supportbot shares status of Zendesk views and number of tickets in Slack when invoked.
