#support chat channel for questions that don't seem appropriate to use the issue tracker or the internal email address for.The overall direction for Support in 2016 is set by our operations strategy, and in order to meet our strategic objective, the following topics are being addressed by the team: (also see the related internal issue )
Increase Capacity
By hiring more service engineers, adding Support Turbos when needed, and making sure that the newly hired service engineers have training materials and pathway readily available and defined when they join. We'll also aim to do a better job acknowledging and incentivizing the rest of the community in their role of supporting the community at large through contributions to documentation and participation in social channels.
Trend discovery
By gathering all our support channels in a central location, ZenDesk, we'll be able to tag tickets with relevant keywords, and gather data on channel load, agent load, and topic trends. Likewise, through implementation of Swiftype on the documentation server, we'll get a better handle on what people are looking for and/or not finding.
Empower Users
By making training materials for customer trainings available on GitLab University, making documentation easy to find and easy to search, and constantly contributing to further documentation with the trend discovery showing where the need is greatest, we can help the community become as self sufficient as possible and empower users to problem-solve. A seamless interface between the support team and developers closes the feedback loop of user experience back to development, and provides 'network' support for the Service Engineers for tough issues.
Improve Customer Experience
By triaging support effort through internal SLA's, the team can improve the support experience for key customers while still addressing the questions of all subscribers, and the rest of the GitLab community.
The Support project hosts an issue tracker meant to improve our workflow by reporting any problems that may arise on our tools or processes. Its also meant to propose and discuss ideas in general.
The issue tracker is open to the community and part of the GitLab.com group. It should not contain any sensitive information. Links to ZenDesk or other references are encouraged.
Every now and then, it may occur that we come close to breaching our SLA's. To prevent an actual breach from occurring, the Support team can call on the help of several 'Support Turbo' developers who are called out on the Team Page. The support team calls for help when necessary via Slack, but in case of doubt or conflicting priorities, the Backend Lead needs to give the green light for the support turbo developers to momentarily switch away from their other tasks.
Certain subscribers have purchased Premium Support (TODO: create /support page), which gives them access to a Dedicated Service Engineer. This means that tickets that arrive in ZenDesk from people within the subscriber's organization are routed to a dedicated SE by way of a trigger in ZenDesk.
The ZenDesk Insight dashboard lists the activity for all our channels currently and summarized over the last 30 days (login to ZenDesk required).
We can't fit more than 13 views in the dashboard panel. We have 6 views that are not channels. Meaning we have 7 views left for all our channels. That is why some channels are combined.
The channels are sorted in order of priority, there are 4 SLA categories:
Preferably we like to answer tickets sooner than the SLA requires. The higher a channel is in the list the sooner it should be answered.
Within ZenDesk, the SLA's are set as follows. Under Business Rules in the Admin console, Service Level Agreements are defined. Currently only one is defined, with different response times required depending on the priority of the ticket which can be Urgent, High, Normal, or Low.
Emergency tickets (SLA category 1 per the list above) are tagged as "Urgent" using a 30 minute max response time setting.
SLA category 2 items are tagged as either High or Normal priority.
An email is sent to everyone in the support team when an SLA breach is imminent, see the automation in ZenDesk.
Within ZenDesk, currently we do not have an SLA assigned (and no automated warnings) to any tickets from channels in category 3 and 4, but we do track response times.
Service Engineers can work on any ticket they feel they can contribute to. However, tickets should be picked up in the following order to make sure that SLA's are not breached, and customers receive the proper level of service:
When a ticket is breaching or has breached its first reply (or next reply) SLA this ticket must be picked up by any Service Engineer independently of who is assigned to it (although see the note re: courtesy in the list above). This also applies to tickets for Premium Customers (i.e. anyone picks up if close to breaching, regardless of who the Dedicated Service Engineer is).
Our service engineers handle the channels listed below. They are sorted in order of priority (strictest SLA at top), and as a result, it is possible that channels that appear lower in this list experience longer delays in receiving responses. We are actively hiring more Service Engineers to strengthen the team and provide support to the community.
When an emergency ticket comes in, it triggers a PagerDuty incident. All Service Engineers must have the PagerDuty application installed on their phones once they are added to the on-call rotation schedule.
When a PD incident is triggered, the alarm will go off for the person on call. You should acknowledge the incident within 5 minutes, or the person on second level support will be alerted. The PD incident will have the link to the corresponding ZenDesk issue from where you will continue the conversation with the customer.
Once acknowledged, you need to login to ZenDesk, go to the corresponding ticket and let the customer know that you will handle their case. On this response you should ask for the best way to contact them. Usual channels are Phone, Skype, WebEx or Hangouts.
If you are unable to help the customer and their instance is in a critical state (unavailable, uncertainty of data loss, etc.), you should escalate the PD incident to second level support, and work through the issue together.
If an emergency takes longer than an hour to resolve, and/or multiple people are or need to be involved, start a google doc that is open to the customer and the wider team at GitLab, and keep track of the issues and ideas there. ZenDesk's 'linear' display of communication with a customer is not as effective in crisis situations, and the majority of developers do not have access to ZenDesk in the first place. Announce the google doc in the appropriate slack channel (#infrastructure, #development, #general) so that individuals can contribute solutions and ideas. When the crisis has been resolved, be sure to transfer pertinent know-how from the google doc to relevant documentation, handbooks, and/or issue trackers, so that the google doc can be deprecated a.s.a.p. In addition, Service Engineers and Developers involved in the crisis should make time to have a hangout for hand-off to make sure that everyone has the chance to recover and stay clear-headed.
When GitLab.com goes down a PD incident will be triggered as well, so the flow is the same as emergency tickets for the first steps. If you are on call and are still working, you should keep an eye on our #infrastructure channel in Slack, as warning messages will often start to appear before the PD incident is triggered.
Once you acknowledge the incident, you need to pinpoint the reason for the outage. Follow the documentation found on our Chef Repo. A fire drill to train you in this situation should take place before you are put on rotation. If this has not happened, please let the team know.
We have a Responsible Disclosure Policy. Emails sent to security@gitlab.com go into ZenDesk and receive an autoresponder that says: "Thank you for your responsible disclosure of a potential GitLab vulnerability. We'll follow up with you within one business day."
Please be very patient with these reports. Do not say 'there is no problem', you might be misunderstanding something that can lead to a 0 day disclosure. Give examples and keep asking questions until you understand the problem or until the researcher concludes there is no problem. If someone invested time to help us, offer to mention them on our Security Researcher Acknowledgments page even if there was no actual vulnerability. If you say that we'll get back to them always mention that they can email us at any time for an update. This is really important to prevent a 0 day disclosure resulting from us forgetting to respond.
If you need help from developers to diagnose the issue please open an issue on dev.gitlab.org so we can work in private. If someone opens a public issue please leave a message: "Thank you for helping to make GitLab more secure! We removed the contents of your vulnerability disclosure to keep it private. We opened an internal issue to look at your disclosure. Can you please use our Responsible Disclosure Policy to send us an email that references this url so we can communicate in private?"
We also use HackerOne to manage security reports. The HackerOne dashboard lists all reports for which you need to respond within one business day. These reports are also piped into ZenDesk, but they need to be responded to from the HackerOne dashboard and closed manually in ZenDesk upon completion. Remember that all researchers should receive feedback as with regular support tickets, and you should not hesitate to triage or escalate the report. Always mention that they can email us at any time for an update. Eventually each report has to be accepted and/or closed through the HackerOne dashboard even if you've followed through on ZenDesk.
After a report has been closed as resolved or informative it can be publicly disclosed. This is can be requested either by GitLab or the reporter. If the reporter requests this then it will be released when you accept it or 30 days after the request is submitted. Prior to publication you should edit the report and make sure that it has:
If you need to grant HackerOne permissions to a new GitLab user, have an admin send an invitation from HackerOne and add you to the Internal group. You can find out who the admins are by asking on the #support channel.
The key used to encode/decode PGP messages is stored in our Support Vault on 1Password. We only provide our public PGP key upon request because it makes collaborating much harder and only a small percentage of all disclosures are serious enough to require that overhead.
See PGP Process for information about using the security PGP key pair and decrypting messages.
You should always answer the tickets in a FIFO manner. Make sure that you answer the tickets that are assigned to you first and then move on to new tickets that have come in and are unassigned, again using FIFO. When you need others to help please create an issue on the relevant GitLab issue tracker.
If GitLab.com users encounter errors on GitLab.com (particularly Error 500s), it may be useful to see the encountered error on our Sentry instance. You can search for a specific username in the search bar by entering in the following:
user.username:
For example:
user.username:sytse
This will bring up all exceptions encountered by the user.
Note that Sentry tries to group similar errors together, but it's possible that it creates a separate entry for the same problem. If a GitLab issue exists for that exception, it will show up by the headline as GL-XYZ, where XYZ is some incident number defined by Sentry. You may have to search other similar events or look on the issue tracker on GitLab.com whether a separate GitLab issue has been created for the Sentry exception.
If the exception looks like it does not exist, feel free to create a GitLab issue by clicking the "Create GitLab Issue" button. This will provide a pre-filled form with all the details of the backtrace. Please clean up the title and/or augment the description with what the user experienced when this exception occurred to make it easier for developers to understand what happened.
We also offer intake, upgrade and installation support for our customers. When these requests come in, you will schedule a call/screen sharing session with them and either guide them through the upgrade process or take control of the server yourself. You should make sure that the customer has a backup before you start the call, as they can take a lot of time to complete and you don't want to do them while in the call. You should also make sure you know as much as possible about the customer's instance as you can.
Important information to collect
We collect this information in ZenDesk and link it to the organization, see the responding to tickets section in onboarding.
While engaging with customers you should always be prepared to jump on a call with them. It is easier to get all the information you might need on a 20 minute call than on 10 2-minute emails. If a conversation goes through several back and forth emails and the problem still isn't close to being resolved, suggest a call via WebEx or Google Hangouts.
If you feel too inexperienced to handle a call, ask someone more experienced to handle the call and listen in if at all possible. After someone else had the call with the customer it is still your responsibility to handle the ticket as long as the ticket is still assigned to you.
Fellow GitLab team members can reach out for help from the Support Team in various ways:
All the comments from our blog are handled by Disqus, and we have an integration in place, where every new comment or reply to a comment creates a new ticket in ZenDesk. As part of your daily support tasks, you should make sure that this queue is as empty as possible. Right now there is no way for us to directly reply to the comment from ZenDesk, so if a comment you see requires a reply (that is if the ticket is not from someone on our team, or is a question or a problem) you need to click on the link to the comment and reply directly in the blog post. Once you have replied, you should come back to ZenDesk and mark that ticket as solved, plus the ticket that was created for your reply.
Tweets that mention @GitLab, @GitLabSupport, or @GitLabStatus, will create a ticket in ZenDesk, and show up in the "Twitter and Facebook" view. All responses should be sent from ZenDesk. If a tweet is responded to from TweetDeck, this risks duplicate responses. Responding from ZenDesk also enables us to track our response times vs. our internal SLA.
Our Developer Advocates also have access to tweets through ZenDesk, and often participate in responding to questions. The support team does not retweet or favorite any tweets; that is handled by the marketing team. Our support team does reply to almost all tweets, following the social media guidelines, regardless of whether the tweet is of a technical nature or not. The marketing team's decision to favorite and/or retweet is independent of the reply provided by the support team; so a tweet may be responded to from ZenDesk, and separately favorited from TweetDeck, as an example.
From time-to-time, we also receive tweets that mention competitors or negative events related to them. In principle, we don't acknowledge these tweets as also described in the developer advocacy section.
The @GitLabStatus account should only be used to give updates on the availability of GitLab.com and to follow up on users reporting that GitLab.com is unavailable.
When a tweet mentions 1 or more of the handles described above, it should be replied to from the main handle (@GitLab).
Tweets use short links which require you to visit that link to make sure you understand the context. To clarify if the request refers to GitLab or an externally hosted GitLab instance as we can only handle requests for gitlab.com.
When resolving Twitter tickets you should:
Messages sent to our Facebook page also feed into ZenDesk.
For ZenDesk issues you will have created issues on the relevant issue tracker. Please refer to the priority as listed under GitLab Workflow in the handbook.
For issues specific to GitLab.com that have nothing to do with availability we have the Support Tracker. This forum must also be checked periodically for new issues and to report back if an issue has been solved. Ensure that you assign the issue to yourself to enable you to keep track of the issue and also to enable other service engineers to easily pick on unassigned tasks at a glance. Some people use this forum to report issues they are having with their on-premises installation. In that case, you should refer them to the CE issue tracker or to our Getting Help page, depending on the issue they are having.
It is always encouraged to take a look at all our issue trackers and respond to bug reports or feature requests:
See the issue triage policies for the above trackers for more information on how issues should be handled.
Many spammers use GitLab.com to create spam. Starting with 8.11, GitLab has better integration with Akismet to check for spam whenever a user creates an issue. This filter needs to be trained properly, however. With some effort our spam detection should improve. If an issue is created on GitLab.com by a spammer, before deleting the account, flag the user's issues as spam:
Once the issues are flagged and the user is verified to be a spammer, go to the spammer's user profile and click on the "Report Abuse" button:
The Akismet filter may also flag false positives and prevent a user from creating issues. The contents of the flagged issue get stored in the spam logs, which is only accessible to admin users. From time to time, we may need to submit the flagged issue as "ham" (e.g. when a user lodges a complaint on the Support Tracker about not being able to create issues). Currently, there are no notifications whenever an entry is flagged, so this needs to be monitored periodically. To flag an issue as ham:
Respond to questions on the GitLab Mailing List.
Questions from the GitLab Forum flow into ZenDesk, but can only be responded to from within the Forum environment.
TODO Questions from Docker's GitLab CE page flow into ZenDesk.
The Stack Overflow tagged questions that relate to GitLab flow into ZenDesk, but can only be responded to from within Stack Overflow.
Respond to mentions of GitLab on Reddit, especially ones in the GitLab Reddit.
Respond to questions about GitLab on Quora, especially the ones that appear in the GitLab Topic channel.
If you have time for it please improve GitLab: fix bugs, add features, and polish the website. You can also consider hanging out on IRC to answer questions and help people (#gitlab on freenode.net).
Two-factor Authentication (2FA) can only be removed from a GitLab.com account if one of the following combinations are met:
Push a certain file to a repository in a personal project
Photo ID and picture of user with ID
Examples:
Support often gets requests from gitlab.com users to release a username. This can be implemented if the following criteria are met:
Usernames will be appended with _idle to keep the account active but release the namespace.
Support agents will not release a current user's contact information to another user. GitLab also will not contact a user on behalf of another user to request surrender of an apparently active username.
Just as service engineers are expected to adhere to the Code of Conduct, we also expect customers to treat service engineers with the same level of respect.
If you receive threatening or hostile emails from a user, please create a confidential issue in the GitLab Support Issue Tracker and include:
Include the Support Lead, VP of Engineering, and Director of PeopleOps in this issue. Together we will evaluate on a case-by-case basis whether to take action (e.g. ban the user from the forums for violating the Code of Conduct).