Support

Communication

Other support pages


Contents of the page


Support Direction

The overall direction for Support in 2016 is set by our operations strategy, and in order to meet our strategic objective, the following topics are being addressed by the team: (also see the related internal issue )

Increase Capacity

By hiring more service engineers, adding Support Turbos when needed, and making sure that the newly hired service engineers have training materials and pathway readily available and defined when they join. We'll also aim to do a better job acknowledging and incentivizing the rest of the community in their role of supporting the community at large through contributions to documentation and participation in social channels.

Trend discovery

By gathering all our support channels in a central location, ZenDesk, we'll be able to tag tickets with relevant keywords, and gather data on channel load, agent load, and topic trends. Likewise, through implementation of Swiftype on the documentation server, we'll get a better handle on what people are looking for and/or not finding.

Empower Users

By making training materials for customer trainings available on GitLab University, making documentation easy to find and easy to search, and constantly contributing to further documentation with the trend discovery showing where the need is greatest, we can help the community become as self sufficient as possible and empower users to problem-solve. A seamless interface between the support team and developers closes the feedback loop of user experience back to development, and provides 'network' support for the Service Engineers for tough issues.

Improve Customer Experience

By triaging support effort through internal SLA's, the team can improve the support experience for key customers while still addressing the questions of all subscribers, and the rest of the GitLab community.

Support Issue Tracker

The Support project hosts an issue tracker meant to improve our workflow by reporting any problems that may arise on our tools or processes. Its also meant to propose and discuss ideas in general.

The issue tracker is open to the community and part of the GitLab.com group. It should not contain any sensitive information. Links to ZenDesk or other references are encouraged.

Support Turbo

Every now and then, it may occur that we come close to breaching our SLA's. To prevent an actual breach from occurring, the Support team can call on the help of several 'Support Turbo' developers who are called out on the Team Page. The support team calls for help when necessary via Slack, but in case of doubt or conflicting priorities, the Backend Lead needs to give the green light for the support turbo developers to momentarily switch away from their other tasks.

Dedicated Service Engineers

Certain subscribers have purchased Premium Support (TODO: create /support page), which gives them access to a Dedicated Service Engineer. This means that tickets that arrive in ZenDesk from people within the subscriber's organization are routed to a dedicated SE by way of a trigger in ZenDesk.

Dashboard

The ZenDesk Insight dashboard lists the activity for all our channels currently and summarized over the last 30 days (login to ZenDesk required).

We can't fit more than 13 views in the dashboard panel. We have 6 views that are not channels. Meaning we have 7 views left for all our channels. That is why some channels are combined.

Service Level Agreements

SLA categories

The channels are sorted in order of priority, there are 4 SLA categories:

  1. Emergency tickets should be answered within 30 minutes.
  2. Security disclosures through regular tickets should be answered within 1 business day.
  3. Disqus and Twitter: we strive to answer them within 1 business day.
  4. Posted Issues through the end: we strive to answer them within two weeks but not sooner than two days, to give the rest of the community a chance to respond.

Preferably we like to answer tickets sooner than the SLA requires. The higher a channel is in the list the sooner it should be answered.

ZenDesk SLA settings and Breach alerts

Within ZenDesk, the SLA's are set as follows. Under Business Rules in the Admin console, Service Level Agreements are defined. Currently only one is defined, with different response times required depending on the priority of the ticket which can be Urgent, High, Normal, or Low.

Emergency tickets (SLA category 1 per the list above) are tagged as "Urgent" using a 30 minute max response time setting.

SLA category 2 items are tagged as either High or Normal priority.

An email is sent to everyone in the support team when an SLA breach is imminent, see the automation in ZenDesk.

Within ZenDesk, currently we do not have an SLA assigned (and no automated warnings) to any tickets from channels in category 3 and 4, but we do track response times.

SLA Workflow

Service Engineers can work on any ticket they feel they can contribute to. However, tickets should be picked up in the following order to make sure that SLA's are not breached, and customers receive the proper level of service:

  1. Tickets that are close to breaching "first time to reply" SLA
  2. Tickets that are close to breaching "next time to reply" SLA
    • It is a courtesy to check with the assigned agent first, before "taking" one of these tickets. Ask if they can respond soon and offer to reply if they don't have the bandwidth.
  3. New tickets (not close to breaching)
  4. Other open tickets

When a ticket is breaching or has breached its first reply (or next reply) SLA this ticket must be picked up by any Service Engineer independently of who is assigned to it (although see the note re: courtesy in the list above). This also applies to tickets for Premium Customers (i.e. anyone picks up if close to breaching, regardless of who the Dedicated Service Engineer is).

Support Channels

Our service engineers handle the channels listed below. They are sorted in order of priority (strictest SLA at top), and as a result, it is possible that channels that appear lower in this list experience longer delays in receiving responses. We are actively hiring more Service Engineers to strengthen the team and provide support to the community.

Emergency Tickets

When an emergency ticket comes in, it triggers a PagerDuty incident. All Service Engineers must have the PagerDuty application installed on their phones once they are added to the on-call rotation schedule.

When a PD incident is triggered, the alarm will go off for the person on call. You should acknowledge the incident within 5 minutes, or the person on second level support will be alerted. The PD incident will have the link to the corresponding ZenDesk issue from where you will continue the conversation with the customer.

Once acknowledged, you need to login to ZenDesk, go to the corresponding ticket and let the customer know that you will handle their case. On this response you should ask for the best way to contact them. Usual channels are Phone, Skype, WebEx or Hangouts.

Crisis Situations

If you are unable to help the customer and their instance is in a critical state (unavailable, uncertainty of data loss, etc.), you should escalate the PD incident to second level support, and work through the issue together.

If an emergency takes longer than an hour to resolve, and/or multiple people are or need to be involved, start a google doc that is open to the customer and the wider team at GitLab, and keep track of the issues and ideas there. ZenDesk's 'linear' display of communication with a customer is not as effective in crisis situations, and the majority of developers do not have access to ZenDesk in the first place. Announce the google doc in the appropriate slack channel (#infrastructure, #development, #general) so that individuals can contribute solutions and ideas. When the crisis has been resolved, be sure to transfer pertinent know-how from the google doc to relevant documentation, handbooks, and/or issue trackers, so that the google doc can be deprecated a.s.a.p. In addition, Service Engineers and Developers involved in the crisis should make time to have a hangout for hand-off to make sure that everyone has the chance to recover and stay clear-headed.

GitLab.com downtime

When GitLab.com goes down a PD incident will be triggered as well, so the flow is the same as emergency tickets for the first steps. If you are on call and are still working, you should keep an eye on our #infrastructure channel in Slack, as warning messages will often start to appear before the PD incident is triggered.

Once you acknowledge the incident, you need to pinpoint the reason for the outage. Follow the documentation found on our Chef Repo. A fire drill to train you in this situation should take place before you are put on rotation. If this has not happened, please let the team know.

Security disclosures

We have a Responsible Disclosure Policy. Emails sent to security@gitlab.com go into ZenDesk and receive an autoresponder that says: "Thank you for your responsible disclosure of a potential GitLab vulnerability. We'll follow up with you within one business day."

Please be very patient with these reports. Do not say 'there is no problem', you might be misunderstanding something that can lead to a 0 day disclosure. Give examples and keep asking questions until you understand the problem or until the researcher concludes there is no problem. If someone invested time to help us, offer to mention them on our Security Researcher Acknowledgments page even if there was no actual vulnerability. If you say that we'll get back to them always mention that they can email us at any time for an update. This is really important to prevent a 0 day disclosure resulting from us forgetting to respond.

If you need help from developers to diagnose the issue please open an issue on dev.gitlab.org so we can work in private. If someone opens a public issue please leave a message: "Thank you for helping to make GitLab more secure! We removed the contents of your vulnerability disclosure to keep it private. We opened an internal issue to look at your disclosure. Can you please use our Responsible Disclosure Policy to send us an email that references this url so we can communicate in private?"

HackerOne

We also use HackerOne to manage security reports. The HackerOne dashboard lists all reports for which you need to respond within one business day. These reports are also piped into ZenDesk, but they need to be responded to from the HackerOne dashboard and closed manually in ZenDesk upon completion. Remember that all researchers should receive feedback as with regular support tickets, and you should not hesitate to triage or escalate the report. Always mention that they can email us at any time for an update. Eventually each report has to be accepted and/or closed through the HackerOne dashboard even if you've followed through on ZenDesk.

After a report has been closed as resolved or informative it can be publicly disclosed. This is can be requested either by GitLab or the reporter. If the reporter requests this then it will be released when you accept it or 30 days after the request is submitted. Prior to publication you should edit the report and make sure that it has:

  1. No sensitive information.
  2. A link to the merge request that fixed the issues.

If you need to grant HackerOne permissions to a new GitLab user, have an admin send an invitation from HackerOne and add you to the Internal group. You can find out who the admins are by asking on the #support channel.

PGP Process

The key used to encode/decode PGP messages is stored in our Support Vault on 1Password. We only provide our public PGP key upon request because it makes collaborating much harder and only a small percentage of all disclosures are serious enough to require that overhead.

See PGP Process for information about using the security PGP key pair and decrypting messages.

Regular ZenDesk tickets

You should always answer the tickets in a FIFO manner. Make sure that you answer the tickets that are assigned to you first and then move on to new tickets that have come in and are unassigned, again using FIFO. When you need others to help please create an issue on the relevant GitLab issue tracker.

Errors on GitLab.com

If GitLab.com users encounter errors on GitLab.com (particularly Error 500s), it may be useful to see the encountered error on our Sentry instance. You can search for a specific username in the search bar by entering in the following:

user.username:

For example:

user.username:sytse

This will bring up all exceptions encountered by the user.

Note that Sentry tries to group similar errors together, but it's possible that it creates a separate entry for the same problem. If a GitLab issue exists for that exception, it will show up by the headline as GL-XYZ, where XYZ is some incident number defined by Sentry. You may have to search other similar events or look on the issue tracker on GitLab.com whether a separate GitLab issue has been created for the Sentry exception.

If the exception looks like it does not exist, feel free to create a GitLab issue by clicking the "Create GitLab Issue" button. This will provide a pre-filled form with all the details of the backtrace. Please clean up the title and/or augment the description with what the user experienced when this exception occurred to make it easier for developers to understand what happened.

Scheduled calls

We also offer intake, upgrade and installation support for our customers. When these requests come in, you will schedule a call/screen sharing session with them and either guide them through the upgrade process or take control of the server yourself. You should make sure that the customer has a backup before you start the call, as they can take a lot of time to complete and you don't want to do them while in the call. You should also make sure you know as much as possible about the customer's instance as you can.

Important information to collect

  1. Type of installation: Source/Omnibus
  2. Current GitLab version
  3. Version you're upgrading to (it isn't always the latest)
  4. Use of GitLab CI (need to upgrade to 8.0 first, then 8.+)

We collect this information in ZenDesk and link it to the organization, see the responding to tickets section in onboarding.

Unscheduled calls

While engaging with customers you should always be prepared to jump on a call with them. It is easier to get all the information you might need on a 20 minute call than on 10 2-minute emails. If a conversation goes through several back and forth emails and the problem still isn't close to being resolved, suggest a call via WebEx or Google Hangouts.

If you feel too inexperienced to handle a call, ask someone more experienced to handle the call and listen in if at all possible. After someone else had the call with the customer it is still your responsibility to handle the ticket as long as the ticket is still assigned to you.

Internal Support

Fellow GitLab team members can reach out for help from the Support Team in various ways:

  1. For normal support questions (can GitLab do x, how do I do x with GitLab) email the internal support email alias that is listed in the "GitLab Email Forwarding" google doc. Those emails create tickets in ZenDesk.
  2. For longer term or larger scope questions, such as discussing customized training requests create an internal issue, adding the "Support" label and @ mentioning one or more Service Engineers.
  3. If customers or users have questions advise them to contact support direct via the relevant channel.
  4. For quick questions that are urgent ping the support team on the support chat channel.

Disqus questions on the blog

All the comments from our blog are handled by Disqus, and we have an integration in place, where every new comment or reply to a comment creates a new ticket in ZenDesk. As part of your daily support tasks, you should make sure that this queue is as empty as possible. Right now there is no way for us to directly reply to the comment from ZenDesk, so if a comment you see requires a reply (that is if the ticket is not from someone on our team, or is a question or a problem) you need to click on the link to the comment and reply directly in the blog post. Once you have replied, you should come back to ZenDesk and mark that ticket as solved, plus the ticket that was created for your reply.

Twitter

Tweets that mention @GitLab, @GitLabSupport, or @GitLabStatus, will create a ticket in ZenDesk, and show up in the "Twitter and Facebook" view. All responses should be sent from ZenDesk. If a tweet is responded to from TweetDeck, this risks duplicate responses. Responding from ZenDesk also enables us to track our response times vs. our internal SLA.

Our Developer Advocates also have access to tweets through ZenDesk, and often participate in responding to questions. The support team does not retweet or favorite any tweets; that is handled by the marketing team. Our support team does reply to almost all tweets, following the social media guidelines, regardless of whether the tweet is of a technical nature or not. The marketing team's decision to favorite and/or retweet is independent of the reply provided by the support team; so a tweet may be responded to from ZenDesk, and separately favorited from TweetDeck, as an example.

From time-to-time, we also receive tweets that mention competitors or negative events related to them. In principle, we don't acknowledge these tweets as also described in the developer advocacy section.

The @GitLabStatus account should only be used to give updates on the availability of GitLab.com and to follow up on users reporting that GitLab.com is unavailable.

When a tweet mentions 1 or more of the handles described above, it should be replied to from the main handle (@GitLab).

Tweets use short links which require you to visit that link to make sure you understand the context. To clarify if the request refers to GitLab or an externally hosted GitLab instance as we can only handle requests for gitlab.com.

When resolving Twitter tickets you should:

  1. Use Play mode in the Twitter view. The default Twitter view will sort tickets by created date (ascending).
  2. Not skip any tickets
  3. Assign the ticket to yourself and ask on the appropriate Slack channel if you don't know the answer
  4. Not cross assign tickets

Facebook

Messages sent to our Facebook page also feed into ZenDesk.

Follow up on issues posted on GitLab issue tracker

For ZenDesk issues you will have created issues on the relevant issue tracker. Please refer to the priority as listed under GitLab Workflow in the handbook.

GitLab.com Support Tracker

For issues specific to GitLab.com that have nothing to do with availability we have the Support Tracker. This forum must also be checked periodically for new issues and to report back if an issue has been solved. Ensure that you assign the issue to yourself to enable you to keep track of the issue and also to enable other service engineers to easily pick on unassigned tasks at a glance. Some people use this forum to report issues they are having with their on-premises installation. In that case, you should refer them to the CE issue tracker or to our Getting Help page, depending on the issue they are having.

GitLab CE/EE/Omnibus issue trackers

It is always encouraged to take a look at all our issue trackers and respond to bug reports or feature requests:

See the issue triage policies for the above trackers for more information on how issues should be handled.

Spam

Flagging Issues as Spam

Many spammers use GitLab.com to create spam. Starting with 8.11, GitLab has better integration with Akismet to check for spam whenever a user creates an issue. This filter needs to be trained properly, however. With some effort our spam detection should improve. If an issue is created on GitLab.com by a spammer, before deleting the account, flag the user's issues as spam:

  1. Login to GitLab.com with an admin account
  2. View each issue and click on "Submit as spam"

Deleting Spammers

Once the issues are flagged and the user is verified to be a spammer, go to the spammer's user profile and click on the "Report Abuse" button:

  1. Login to GitLab.com with an admin account
  2. Go to: https://gitlab.com/admin/abuse_reports
  3. Be careful: Click on "Delete user" only if you are sure this user is a spammer.

False positives: Ham

The Akismet filter may also flag false positives and prevent a user from creating issues. The contents of the flagged issue get stored in the spam logs, which is only accessible to admin users. From time to time, we may need to submit the flagged issue as "ham" (e.g. when a user lodges a complaint on the Support Tracker about not being able to create issues). Currently, there are no notifications whenever an entry is flagged, so this needs to be monitored periodically. To flag an issue as ham:

  1. Login to GitLab.com with an admin account
  2. Go to: https://gitlab.com/admin/spam_logs
  3. Scan the contents of each log. If it appears that a user is trying to post valid information, click on "Submit as ham".

Mailing list

Respond to questions on the GitLab Mailing List.

Forum

Questions from the GitLab Forum flow into ZenDesk, but can only be responded to from within the Forum environment.

TODO Docker

TODO Questions from Docker's GitLab CE page flow into ZenDesk.

Stack Overflow

The Stack Overflow tagged questions that relate to GitLab flow into ZenDesk, but can only be responded to from within Stack Overflow.

Reddit

Respond to mentions of GitLab on Reddit, especially ones in the GitLab Reddit.

Quora

Respond to questions about GitLab on Quora, especially the ones that appear in the GitLab Topic channel.

Non channel work

If you have time for it please improve GitLab: fix bugs, add features, and polish the website. You can also consider hanging out on IRC to answer questions and help people (#gitlab on freenode.net).

Support Policies

Removal of 2FA

Two-factor Authentication (2FA) can only be removed from a GitLab.com account if one of the following combinations are met:

Combination 1:

Push a certain file to a repository in a personal project

Combination 2:

Photo ID and picture of user with ID

Examples:

Dormant Usernames

Support often gets requests from gitlab.com users to release a username. This can be implemented if the following criteria are met:

  1. The account in question has no data.
  2. There hasn't been any activity on the account (i.e. code, issues, merge requests) for six months. This considers activity on projects the account owns as well as activity on projects the account participants in.
  3. Support has had two failed attempts to contact the user within a two week period.
  4. The user cannot be an owner of any active project.

Usernames will be appended with _idle to keep the account active but release the namespace.

Support agents will not release a current user's contact information to another user. GitLab also will not contact a user on behalf of another user to request surrender of an apparently active username.

Code of Conduct

Just as service engineers are expected to adhere to the Code of Conduct, we also expect customers to treat service engineers with the same level of respect.

If you receive threatening or hostile emails from a user, please create a confidential issue in the GitLab Support Issue Tracker and include:

  1. A history of the user's communication
  2. Relevant links
  3. Summary of the high-level issues

Include the Support Lead, VP of Engineering, and Director of PeopleOps in this issue. Together we will evaluate on a case-by-case basis whether to take action (e.g. ban the user from the forums for violating the Code of Conduct).