Support Operations team is responsible for supporting the day-to-day operations and software systems used by GitLab’s global Technical Support team, primarily our Zendesk instance, and integrations with internal business processes and tools. The Support Operations Team is exclusively responsible for handling Zendesk Support and Explore related queries.
You can locate us in GitLab issues or via the
#support_operations
channel in slack.
To help the Technical Support Team spontaneously and make their day to day workflow simple and easier so the efficiency and result values of Technical Support Team increases.
To provide the “Best in Class” support to our support team specifically and GitLab to increase the overall value of GitLab as a team, organization and product.
Vision Target 2021- 2022:
| Name | Role |
|---|---|
| Lyle Kozloff | Director of Support, Global Readiness |
| Jason Colyer | Support Operations Manager |
| Nabeel Bilgrami | Support Operations Specialist (Global Instance) |
| Alyssa Villa | Support Operations Specialist (Global Instance) |
| Dylan Tragjasi | Support Operations Specialist (Global Instance) |
| Sarah Cole | Support Operations Specialist (Global Instance) |
This diagram explains how the general fixes are handled by the Support Operations team.
The Support Operations Team usually make the critical changes over weekends
and communicate them via Slack channel #support_ops-annoucements
and SWIR (/handbook/support/#support-week-in-review).
Mirroring infrastructure's change management criticalities Support Ops defines changes on a C1 - C4 scale that helps determine appropriate planning horizons.
Criticalities can be estimated during a Requested Change and will be finalized once an issue is moved to the Support Ops project. The scheduling intervals indicate the delay between change annoucement and change deployment.
These are changes with high impact or high risk that may significantly modify Support Engineer or Customer experience. If a change is going to cause downtime to the environment, it is always categorized a C1.
These are changes that aren't expected to significantly impact Support Engineer or Customer experiences, but which still carry some risk of impact if something unexpected happens.
These are changes with either no or very-low risk of negative impact, but where there is still some inherent complexity, or it is not fully automated and hands-off.
These are changes that are exceedingly low risk and commonly executed, or which are fully automated. Often these will be changes that are mainly being recorded for visibility rather than as a substantial control measure.
Each criticiality has a required time interval that ensures adequate time for Ops scheduling and for the change creator to go through an appropriate change management process. If the change is of high priority, the required interval can be overriden pending the approval at the appropriate level.
| Criticality | Horizon | Approval |
|---|---|---|
| C1 | 4 weeks | VP of Customer Support |
| C2 | 3 weeks | Director of Support Readiness |
| C3 | 2 weeks | Support Ops Manager |
| C4 | No horizon required | n/a - changes can be scheduled immediately |
To help ensure the team doesn't get overwhelmed and has the ability to focus on specialization and learning, we divide out the responsibilties amongst our team. The current division of responsibilities is:
| Category | Area | Primary DRI | Secondary DRI |
|---|---|---|---|
| Access Requests | Internal License Requests | Everyone | Everyone |
| Zendesk Global Provisioning | @jcolyer | @lyle | |
| Zendesk US Federal Provisioning | @jcolyer | @lyle | |
| Account Deletions | Form | @jcolyer | @dtragjasi |
| Issues | @jcolyer | @avilla4 | |
| Processor | @jcolyer | @avilla4 | |
| Questions | @jcolyer | @secole | |
| Audits | 1Password | @jcolyer | @lyle |
| Calendly | @avilla4 | @jcolyer | |
| GitLab.com | @jcolyer | @lyle | |
| Pagerduty | @dtragjasi | @jcolyer | |
| Zendesk Global | @nabeel.bilgrami | @jcolyer | |
| Zendesk US Federal | @secole | @jcolyer | |
| Calendly | Management | @avilla4 | @dtragjasi |
| Procurement | @jcolyer | @lyle | |
| Provisioning | @jcolyer | @lyle | |
| Forms | Customer Ticket Generator | @nabeel.bilgrami | @avilla4 |
| Free User Offer Form | @jcolyer | @dtragjasi | |
| L&R Internal Requests Form (Global) | @jcolyer | @dtragjasi | |
| L&R Internal Requests Form (US Federal) | @jcolyer | @dtragjasi | |
| US Federal Customer Feedback | @secole | @jcolyer | |
| Usage Ping Requests | @jcolyer | @lyle | |
| Other Software | Customer Feedback Processor US Federal | @jcolyer | @secole |
| Free user offer processor | @jcolyer | @dtragjasi | |
| L&R Internal Request Processor (Global) | @jcolyer | @secole | |
| L&R Internal Request Processor (US Federal) | @jcolyer | @secole | |
| 1-1-issue-generator | @nabeel.bilgrami | @jcolyer | |
| support/team page | Everyone | Everyone | |
| Pagerduty | Management | @avilla4 | @jcolyer |
| Provisioning | @jcolyer | @lyle | |
| Slack | SGG Slackbot | @avilla4 | @jcolyer |
| Workflows | @jcolyer | @nabeel.bilgrami | |
| Status.io | Provisioning | @jcolyer | @lyle |
| Support Ops Project | Everyone | Everyone | |
| Ticket work | Everyone | Everyone | |
| Zapier | Support Ops Zap management | @jcolyer | @secole |
| Zendesk Global | Agents | @avilla4 | @dtragjasi |
| Articles | @dtragjasi | @jcolyer | |
| Automations | @nabeel.bilgrami | @avilla4 | |
| Contacts | @jcolyer | @nabeel.bilgrami | |
| Forms and Fields | @jcolyer | @nabeel.bilgrami | |
| Explore | @nabeel.bilgrami | @avilla4 | |
| Macros | Everyone | Everyone | |
| Organizations | @dtragjasi | @avilla4 | |
| Priority Prospects | @jcolyer | @secole | |
| Procurement | @jcolyer | @lyle | |
| Salesforce Cases | @jcolyer | @nabeel.bilgrami | |
| SLAs | @jcolyer | @lyle | |
| Ticket Processor | @jcolyer | @nabeel.bilgrami | |
| Ticket Round Robin | @nabeel.bilgrami | @dtragjasi | |
| Triggers | @nabeel.bilgrami | @avilla4 | |
| Views | @avilla4 | @nabeel.bilgrami | |
| Wizard | @jcolyer | @dtragjasi | |
| Zendesk Apps | @dtragjasi | @jcolyer | |
| Zendesk Theme | @dtragjasi | @jcolyer | |
| Zendesk-Salesforce Sync | @jcolyer | @secole | |
| Zendesk US Federal | Agents | @secole | @jcolyer |
| Automations | @secole | @jcolyer | |
| Macros | @secole | @jcolyer | |
| Procurement | @jcolyer | @lyle | |
| Salesforce Cases | @jcolyer | @secole | |
| Ticket forms and Fields | @secole | @jcolyer | |
| Ticket Round Robin | @jcolyer | @secole | |
| Triggers | @secole | @jcolyer | |
| Views | @secole | @jcolyer | |
| Zendesk Apps | @secole | @jcolyer | |
| Zendesk Theme | @secole | @jcolyer | |
| Zendesk-Salesforce Sync | @jcolyer | @secole |
During the month of December, Support Operations enters a code freeze. During this period, Support Operations will not deploy any major changes to the various systems we manage (Zendesk, Calendly, etc.).
For reference, a "major" change would be anything impacting ticket routing or Support workflows. Some general examples would be:
This is done to promote stability, especially during a time many of us take time off for the holidays. We also use this time to review our setup and look for any unnoticed problems/errors/etc.
During this time, we will still do the following:
Currently, Support Operations is using a ratios for our hiring plan. The ratios used are:
There are plans to utilize the information at the Support Ops Metrics page in conjuction with the above ratios to refine this hiring plan in the future. The Support Ops Metrics page can be used to get a quick look at how we are currently doing and help determine future needs.
If we receive any problem in using Zendesk, can we contact Zendesk directly?
Please contact Support-Ops team first. Discuss the problem by asking a question in channel and tagging @support-ops. It is a high probability that we can help you resolve the problem at hand. In cases where we cannot and we do need to contact Zendesk support directly, it is best to have Support-Ops handle that.
What will happen if Zendesk is down globally?
Zendesk will only go down when the internet is globally effected because they use Pods for services. This ensures that if a region is facing downtime, Zendesk can quickly mitigate that while making sure services run smoothly. However, if you are still facing any problem accessing Zendesk, please contact the support-ops team. In the case that Zendesk is down globally, we have email support option available.
Is there any disaster recovery plan available?
Zendesk keeps the data in backup servers will all due diligence. This ensures that we can recover data when it is needed. These backups are utilized to restore Zendesk in the case it fails due to a problem on Zendesk's end.
Also, the support-ops team ensures all triggers, automations, views, macros, forms, fields, conditions, etc are documented to save the hassle of writing up everything from scratch.
Why do we allow users to open support tickets without being required to login to Zendesk via some authentication?
According to Lee Matos:
Circa 2016 we had open Zendesk where we were manually assigning users to manually created orgs.
There was a project that was being created called “middleware” that was going to act as the SSOT of known orgs. It did nothing to solve the customer mapping.
It was in scope creep mode so I explored the offical sfdc<->zd sync. We started with account sync. We were interested in user syncing but our data was a mess.
We ran into problems with org sync. Solved a bunch then some more.
We recreated sfdc sync for the Global Support instance and US Federal instance and are still just syncing orgs.
There is a push (rightfully so) to lock it down and improve our wall/first time unknown user flow. If someone can get that over the line I think that GitLab support engineering leadership has no good reason to keep it “open access” at the moment and we support changing it. (I'm speaking out of turn — I’m stating this as fact but it’s my opinion.)
