The Support Operations team maintains the day-to-day operations and software systems used by GitLab’s global Technical Support team.
At GitLab, SSAT is a measure of how satisfied our customers are with their interaction with the GitLab Support team. This is based on survey responses from customers after each ticket is solved by the Support team, using a Good or Bad rating.
Note: We use the term SSAT, but another common term for what we do is CSAT (Customer Satisfaction).
The target is 95% satisfaction. The definition of the KPI is:
Satisfied [total good scores] / Total Survey Responses [good scores + bad scores]
This calculation excludes cases where a survey was not offered or where it was offered but no score was provided.
The survey is sent out 24 hours after a ticket is solved. The tag csat_survey_sent is added to a ticket as soon as a survey is sent out.
There are a few reasons why a customer wouldn’t receive a Survey after a ticket has been solved:
close_by_merge).
Starting July 9th, 2019, in order to learn about the reasons behind a Bad satisfaction rating, we ask any customer who gives a negative Survey response to select a reason for their dissatisfaction. This feedback is shared with the Product team and the Support Managers, depending on which reason is chosen. The dropdown options are:
All negative and positive feedback with comment is exported to an internal repository. Once the feedback has been submitted, an Issue is created and the Support Engineering Managers are tagged when a negative rating has been submitted. It is the responsibility of the Support Engineering Managers to investigate the feedback and ensure appropriate actions are taken to either resolve the root cause of the negative experience or reduce the likelihood of it recurring.
We have the option of allowing all of the end-users in an organization to see each other's tickets. This is referred to as a shared organization in Zendesk.
We can grant the end-users permissions to see the tickets at the organization level and the personal level (personal level overrides org level). This means that at the personal level, we can allow selected individuals in an org to access all the tickets created by members of their org or we can allow everyone in the company to see everyone's tickets.
Shared organizations will only be set up if we receive a request directly from the customer through a Support Ticket. Once the ticket is created, any member of the Customer Success or Support teams can create a request in the Support Ops project using the Shared Organization Request template, and any member of the Support Team can update the organization settings once the issue is created.
There are a few different options available for our customers:
Note: If the organization-wide sharing setting has been disabled, the setting on the individual user’s account will override the organization-wide setting, and the end user will have full access to tickets from the organization. If the organization-wide sharing setting has been enabled, the setting on the individual end user is redundant.
We have enabled Zendesk to send all ticket information to Salesforce using a target. Currently, we create a Salesforce Zendesk Ticket record whenever a ticket is created created and when it is either:
Once the information is received in Salesforce, a notification is sent from SFDC to the Technical Account Managers so they are aware of the issues their accounts are experiencing.
To troubleshoot any issues with this functionality, we need to make sure that:
Use auto-matching criteria for searching Salesforce recordsZendesk Support Ticket and Zendesk Support Ticket Comment is visible to the System AdministratorWhen working on the Salesforce side of this configuration, make sure to always communicate with the Sales Systems and Operations team about any inconsistencies you find and request their help to troubleshoot any issues.
To ensure the data in SFDC is synced with Zendesk, we utilize two methods:
Currently, it syncs based on the Salesforce ID (18) value for all accounts that
have a type of Customer or Former Customer and do not meet the US Federal
criteria.
Accounts are synced with Organizations as long as:
Salesforce ID (18) in SFDC matches the one Salesforce ID in ZendeskWe have configured the following to sync:
If you encounter issues with the SFDC<>Zendesk sync, please create an issue via the support-ops-project.
This is handled via a scheduled pipeline running from the ops instance. It
handles the syncing of Federal Accounts and Contacts from SFDC. This runs every
4 hours. It syncs based on the Salesforce ID (18) value. As per Zendesk rules,
every Account Name and Contact Email must be unique so that it can be synced.
The criteria for the federal sync determines what is and is not synced. Currently, it must meet all of the following criteria:
Customer, Former Customer, or ProspectSupport Level is either Premium or Ultimate.Support Level is Basic and the Number of Licenses (This Account) is
2000 or more.If you encounter issues with the SFDC<>US Federal Zendesk sync, please create an issue via the support-ops-project.
PagerDuty is how we dispatch emergencies and handle scheduling for different roles.
A service in PagerDuty usually represents a component a team monitors. For Support's purposes, however, it's just
something you'd go on call for. Services interact with /chatops oncall commands to display who is currently on-call
in Slack.
The Customer Support Service is the service we use to track who is on-call for Emergency Support.
For more information, read more about how being on-call works.
This service is driven by Support Engineers participating in the Customer Emergency Rotation
An incident can be generated by:
When an incident occurs, it will alert in #support_self-managed on Slack.
The Customer Supoprt - US Federal Service is the service we use to track who is on-call for Emergency Support for US Federal Customers.
This service is driven by US Citizens participating in the US Federal Customer Emergency Rotation
An incident can be generated by:
When an incident occurs, it will not alert in Slack.
The Incident Management - CMOC Service is the service we use to track who is on-call for the role of Communications Manager on Call during GitLab.com operational emergencies.
This service is driven by Support Engineers participating in the Incident Management - CMOC Rotation.
An incident can be generated by:
The FRT Hawk Service is what we use to track who is responsible for playing the role of FRT Hawk.
This service is driven by Support Engineers participating in the Customer Support - FRT Hawk Rotation.
This Service has no integrations and is used only for scheduling.
The SLA Hawk Service is what we use to track who is responsible for playing the role of SLA Hawk.
This service is driven by Support Engineers participating in the Customer Support - SLA Hawk Rotation.
This Service has no integrations and is used only for scheduling.
The Support Managers Service is what we use to track which Support Manager is on-call.
For more information, read about what a Support Manager On-call does
This service is driven by Support Manager participating in the Support Managers Rotation.
An incident can be generated by:
/pd-support-manager in SlackWhen an incident occurs, it will alert in #support_managers in Slack.
The SSAT Reviewing Manager Service is a tracking only rotation indicating the Support Manager who will review the SSAT feedback for that week.
For more information, read about Responding to SSAT Feedback
An escalation policy is built out of PagerDuty Schedules with rules that define when things should be moved to the next layer.
The Customer Emergency Rotation defines who should be paged in the event of a customer emergency.
Immediately after an incident is triggered, notify:
- Customer Emergencies - AMER
- Customer Emergencies - APAC
- Customer Emergencies - EMEA
Escalate after 10 minutes:
- Support Manager - AMER
- Support Manager - APAC
- Support Manager - EMEA
Escalate after 5 minutes:
- Customer Emergencies EMEA Escalation
- Drew, Arihant, Harish, Cindy, Athar, Aric, Davin, Adam
Repeat 5 times if no one acknowledges incidents
The US Federal Customer Emergency Rotation defines who should be paged in the event of a US Federal customer emergency.
Immediately after an incident is triggered, notify:
- US Federal On-Call
Escalate after 15 minutes:
- Support Manager - AMER
Escalate after 20 minutes:
- Lyle, Lee, Jason
The Incident Management - CMOC Rotation defines who should be paged in the event of a GitLab.com incident.
Immediately after an incident is triggered, notify:
- Incident Management - CMOC (AMER)
- Incident Management - CMOC (APAC)
- Incident Management - CMOC (EMEA)
Escalate after 5 minutes
- Support Manager - AMER
- Support Manager - EMEA
- Support Manager - APAC
- SRE Managers Americas
- SRE Managers EMEA
- SRE Managers Weekends
Escalate after 5 minutes:
- Anthony, Dave, Brent (Infrastructure)
The Customer Support - FRT Hawk Escalation Policy defines who is playing the role of FRT Hawk.
It shouldn't ever page, as it's used only to facilitate scheduling.
Immediately after an incident is triggered, notify:
- FRT Hawk - AMER Easterly
- FRT Hawk - AMER Westerly
- Customer Emergencies - APAC
- Customer Emergencies - EMEA
The Customer Support - SLA Hawk Escalation Policy defines who is playing the role of SLA Hawk.
It shouldn't ever page, as it's used only to facilitate scheduling.
Immediately after an incident is triggered, notify:
- SLA Hawk - AMER Easterly
- SLA Hawk - AMER Westerly
The Support Managers Escalation Policy defines which Support Manager is on call.
Immediately after an incident is triggered, notify:
- Support Manager - AMER
- Support Manager - EMEA
- Support Manager - APAC
PagerDuty schedules drive the actual rotations of individuals and track overrides.
For ease, we define the hours of on-call duty per region as follows:
Hand-off: Monday at 07:00 UTC.
At writing (2019-10-04) there are no further subdivisions per region.
The following schedules are used:
Customer Emergencies
US Federal Customer Emergencies
US Federal Customer Emergencies is split into two layers:
FRT Hawks
SLA Hawks
Support Managers
You can subscribe to a WebCal feed, suitable for viewing in Google Calendar.
From any of the above links you can subscribe to the whole schedule by clicking Schedule Info and then WebCal feed. Alternatively you can pull just your schedule from general Schedules page by clicking Export then Just my calendar in the WebCal Feed section.
If you want just one calendar for all of your on-call, you can grab a WebCal feed by clicking on your profile picture, going to My Profile and then On-Call Shift, then clicking the Export button to reveal the link.
The premium breach notifications is a webhook that is triggered by a ZenDesk automation titled Premium important ticket slack notification. The trigger looks for any
high premium tagged tickets that have less than 2 hours left on the breach clock.
The automation runs at the top of every hour
and due to this limitation, it does not run at the exact hour the ticket is less than 2 hour from breaching.
Once the ticket is updated, a trigger sends a webhook to slack which is configured on the slack side.
The #feed_zd-self-managed channel is updated by a webhook that's triggered whenever a Self-Managed ticket is created or updated. The Zendesk trigger is called "Slack Support Live Feed (With Organization names)"
The #feed_zd-self-managed channel does not display tickets addressed to GitLab.com or
GitHost.io.
The #feed_zd_gitlab-com channel is updated by a webhook that's triggered whenever a GitLab.com ticket is created or updated. The Zendesk trigger is called "Slack Services Live Feed". This channel does not display EE tickets.
The Supportbot shares status of Zendesk views and number of tickets in Slack when invoked.
