Gitlab hero border pattern left svg Gitlab hero border pattern right svg

Support Engineer Ops

On this page

The Support Operations team maintains the day-to-day operations and software systems used by GitLab’s global Technical Support team.

Customer Satisfaction Survey (CSAT)

At GitLab, CSAT is a measure of how satisfied our customers' are with their interaction with the GitLab Support team. This is based on survey responses from customers after each ticket is solved by the Support team using a Good or Bad rating.

The target is 95% customer satisfaction. The definition of the KPI is:

Satisfied [total good scores]  / Total Survey Responses [good scores + bad scores]

This calculation excludes cases where a survey was not offered or where it was offered but no score was provided.

The survey is sent out 24 hours after a ticket is solved. The tag csat_survey_sent is added to a ticket as soon as a survey is sent out.

CSAT in numbers

There are a few reasons why a customer wouldn’t receive a CSAT Survey after a ticket has been solved:

Managing Negative Feedback

Starting July 9th, 2019, in order to learn about the reasons behind a bad satisfaction rating, we ask any customer who gives a negative survey response to select a reason for their dissatisfaction. This feedback is shared with the Product team and the Support Managers, depending on which reason is chosen. The dropdown options are:

All negative and positive feedback with comment is exported to an internal repository. Once the feedback has been submitted, an Issue is created and the Support Engineering Managers are tagged when a negative rating has been submitted. It is the responsibility of the Support Engineering Managers to investigate the feedback and ensure appropriate actions are taken to either resolve the root cause of the negative experience or reduce the likelihood of it recurring.

Shared Organizations in Zendesk

We have the option of allowing all of the end-users in an organization to see each other's tickets. This is referred to as a shared organization in Zendesk.

We can grant the end-users permissions to see the tickets at the organization level and the personal level (personal level overrides org level). This means that at the personal level, we can allow selected individuals in an org to access all the tickets created by members of their org or we can allow everyone in the company to see everyone's tickets.

Shared organizations will only be set up if we receive a request directly from the customer through a Support Ticket. Once the ticket is created, any member of the Customer Success or Support teams can create a request in the Support Ops project using the Shared Organization Request template, and any member of the Support Team can update the organization settings once the issue is created.

There are a few different options available for our customers:

Salesforce - Zendesk sync

Ticket sending to Salesforce

We have enabled Zendesk to send all ticket information to Salesforce using a target. Currently, we create a Salesforce Zendesk Ticket record whenever a ticket is created and when it is solved by one of our agents or an automation. Once the information is received in Salesforce, a notification is sent from SFDC to the Technical Account Managers so they are aware of the issues their accounts are experiencing.

To troubleshoot any issues with this functionality, we need to make sure that:

When working on the Salesforce side of this configuration, make sure to always communicate with the Sales Systems and Operations team about any inconsistencies you find and request their help to troubleshoot any issues.

Account - Organization sync from Salesforce

All organizations in Zendesk are created through our Salesforce Integration. Currently, only records with Account Type equal to Customer are synced from Salesforce to Zendesk.

Accounts are synced with Organizations as long as the Account ID in SFDC matches the one Salesforce ID in Zendesk and the Zendesk Support Organization ID in SFDC matches the Zendesk Organization ID in Zendesk.

We have configured custom field mappings:

A script was created by the Support Team in order to do routine checks of data integrity between Salesforce and Zendesk. This script runs daily and it checks for:

Other tools

PagerDuty

PagerDuty is how we dispatch emergencies and handle scheduling for different roles.

PagerDuty Services

A service in PagerDuty usually represents a component a team monitors. For Support's purposes, however, it's just something you'd go on call for. Services interact with /chatops oncall commands to display who is currently on-call in Slack.

Customer Support

The Customer Support Service is the service we use to track who is on-call for Emergency Support.

For more information, read more about how being on-call works.

This service is driven by Support Engineers participating in the Customer Emergency Rotation

An incident can be generated by:

When an incident occurs, it will alert in #support_self-managed on Slack.

Customer Support - US Federal

The Customer Supoprt - US Federal Service is the service we use to track who is on-call for Emergency Support for US Federal Customers.

This service is driven by US Citizens participating in the US Federal Customer Emergency Rotation

An incident can be generated by:

When an incident occurs, it will not alert in Slack.

FRT Hawk

The FRT Hawk Service is what we use to track who is responsible for playing the role of FRT Hawk.

This service is driven by Support Engineers participating in the Customer Support - FRT Hawk Rotation.

This Service has no integrations and is used only for scheduling.

SLA Hawk

The SLA Hawk Service is what we use to track who is responsible for playing the role of SLA Hawk.

This service is driven by Support Engineers participating in the Customer Support - SLA Hawk Rotation.

This Service has no integrations and is used only for scheduling.

Support Managers

The Support Managers Service is what we use to track which Support Manager is on-call.

For more information, read about what a Support Manager On-call does

This service is driven by Support Manager participating in the Support Managers Rotation.

An incident can be generated by:

When an incident occurs, it will alert in #support-managers in Slack.

PagerDuty Escalation Policies

An escalation policy is built out of PagerDuty Schedules with rules that define when things should be moved to the next layer.

Customer Emergency Rotation

The Customer Emergency Rotation defines who should be paged in the event of a customer emergency.

Immediately after an incident is triggered, notify:

  • Customer Emergencies - AMER
  • Customer Emergencies - APAC
  • Customer Emergencies - EMEA

Escalate after 10 minutes:

  • Support Manager - AMER
  • Support Manager - APAC
  • Support Manager - EMEA

Escalate after 5 minutes:

  • Customer Emergencies EMEA Escalation
  • Drew, Arihant, Harish, Cindy, Athar, Aric, Davin, Adam

Repeat 5 times if no one acknowledges incidents

US Federal Customer Emergency Rotation

The US Federal Customer Emergency Rotation defines who should be paged in the event of a US Federal customer emergency.

Immediately after an incident is triggered, notify:

  • US Federal On-Call

Escalate after 15 minutes:

  • Support Manager - AMER

Escalate after 20 minutes:

  • Lyle, Lee, Jason
Customer Support - FRT Hawk

The Customer Support - FRT Hawk Escalation Policy defines who is playing the role of FRT Hawk.

It shouldn't ever page, as it's used only to facilitate scheduling.

Immediately after an incident is triggered, notify:

  • FRT Hawk - AMER Easterly
  • FRT Hawk - AMER Westerly
  • Customer Emergencies - APAC
  • Customer Emergencies - EMEA
Customer Support - SLA Hawk

The Customer Support - SLA Hawk Escalation Policy defines who is playing the role of SLA Hawk.

It shouldn't ever page, as it's used only to facilitate scheduling.

Immediately after an incident is triggered, notify:

  • SLA Hawk - AMER Easterly
  • SLA Hawk - AMER Westerly
Support Managers

The Support Managers Escalation Policy defines which Support Manager is on call.

Immediately after an incident is triggered, notify:

  • Support Manager - AMER
  • Support Manager - EMEA
  • Support Manager - APAC

PagerDuty Schedules

PagerDuty schedules drive the actual rotations of individuals and track overrides.

Regional hours of operation

For ease, we define the hours of on-call duty per region as follows:

Hand-off: Monday at 07:00 UTC.

At writing (2019-10-04) there are no further subdivisions per region.

The following schedules are used:

Customer Emergencies

US Federal Customer Emergencies

FRT Hawks

SLA Hawks

Support Managers

Subscribing to a Schedule

You can subscribe to a WebCal feed, suitable for viewing in Google Calendar.

From any of the above links you can subscribe to the whole schedule by clicking Schedule Info and then WebCal feed. Alternatively you can pull just your schedule from general Schedules page by clicking Export then Just my calendar in the WebCal Feed section.

If you want just one calendar for all of your on-call, you can grab a WebCal feed by clicking on your profile picture, going to My Profile and then On-Call Shift, then clicking the Export button to reveal the link.

Slack notifications

Premium Breach Notifications

The premium breach notifications is a webhook that is triggered by a ZenDesk automation titled Premium important ticket slack notification. The trigger looks for any high premium tagged tickets that have less than 2 hours left on the breach clock. The automation runs at the top of every hour and due to this limitation, it does not run at the exact hour the ticket is less than 2 hour from breaching.

Once the ticket is updated, a trigger sends a webhook to slack which is configured on the slack side.

#zd-self-managed-feed and #zd-gitlab-com-feed

The #zd-self-managed-feed channel is updated by a webhook that's triggered whenever a Self-Managed ticket is created or updated. The Zendesk trigger is called "Slack Support Live Feed (With Organization names)" The #zd-self-managed-feed channel does not display tickets addressed to GitLab.com or GitHost.io.

The #zd-gitlab-com-feed channel is updated by a webhook that's triggered whenever a GitLab.com ticket is created or updated. The Zendesk trigger is called "Slack Services Live Feed". This channel does not display EE tickets.

Support Bot

The Supportbot shares status of Zendesk views and number of tickets in Slack when invoked.