This is a Controlled Document
Inline with GitLab's regulatory obligations, changes to controlled documents must be approved or merged by a code owner. All contributions are welcome and encouraged.
GitLab takes the security of our clients’ information extremely seriously, regardless of whether it’s on GitLab.com or in a self-managed instance. In keeping with GitLab’s value of transparency we believe in communicating about security incidents clearly and promptly.
This communication response plan maps out the who, what, when, and how of GitLab in notifying and engaging with internal stakeholders and external customers on security incidents. This plan of action covers the strategy and approach for security events which have a ‘high’ or greater impact as outlined in GitLab’s risk scoring matrix.
|GitLab Team Members||Responsible for following the requirements in this procedure|
|SIRT||Responsible for implementing and executing this procedure|
|SIRT Management (Code Owners)||Responsible for approving significant changes and exceptions to this procedure|
The GitLab Security team identifies security incidents as any violation, or threat of violation, of GitLab security, acceptable use or other relevant policies. You can learn more about how we identify incidents in the GitLab security incident response guide.
For Support or Infrastructure managed incidents where external communication guidance is needed, please see the corporate communications incident response plan and engage that team via #corp-comms in slack.
For Infrastructure incidents, please follow the infrastructure incident management and communication process.
Security Engineer On-Call will determine the scope, severity and potential impact of the security incident. Once the potential impact has been determined, implementation of the appropriate internal and external communications strategy should begin.
|Security Engineer on Call (SEOC):||This is the on-call Security Operations Engineer. The individual is the first to act, validate, and begin the process of determining severity and scope.|
|Security Incident Manager on Call (SIMOC):||This is a Security Engineering Manager who is engaged when incident resolution requires coordination across multiple parties. The SIMOC is the tactical leader of the incident response team, typically not engaged to perform technical work. The SIMOC assembles the incident team by engaging individuals with the skills, access, and information required to resolve the incident. The focus of the SIMOC is to keep the incident moving towards resolution, keeping stakeholders informed and performing SecCMOC duties.|
|Security Communications Manager on Call (SecCMOC):||This is the Security Incident Manager On-Call (SIMOC), Security Incident Commander or Security Assurance Engineer who will coordinate external communications efforts according to this security incident response plan and liaise across the extended GitLab teams to ensure all parties are activated, updated and aligned.|
|Security External Communications:||This function partners with and advises incident response teams in reviewing and improving messaging for external audiences (including customers, media, broader industry). This role laises with marketing teams for any necessary reviews or messaging deployment. This function should be engaged once first draft content has been developed using the Security incident external response issue template.|
As security practitioners and incident response engineers, our security assurance and security operations teams and engineers are best positioned to develop initial messaging and serve in the
Communications manager on call role.
CMOC is the DRI for:
Security External Communications function is the DRI for:
#customer-success) for awareness and use.
Support manager on callwill manage support team awareness
incident_communicationstemplate, tagging in the assigned timezone's
coverage ownerand posting the issue in the #mktgops channel in Slack.
Support Team: Using background information and prepared responses provided by the Security Engineer On Call and Communications Manager On Call, our Support Team will triage and respond to customer communications stemming from the security incident. Contact the on-call manager via
#spt_managers in Slack. If it's urgent page the
Support Manager On-call using
/pd-support-manager command in slack. To ensure this group has early awareness on security incidents and events they are autotagged as an FYI in the security-external-incident-or-event-response template.
Community Relations: May need to respond to customers and the general public via social channels, as such should be engaged before public-facing materials are released. Any prepared responses or FAQs should be provided to assist with their interactions. Contact this group in
#developer-relations or any Slack channel by pinging
@community-team. To ensure this group has early awareness on security incidents and events they are autotagged as an FYI in the
Security incidents can be high-pressure, high-stress situations. Everyone is anxious to understand the details around the investigation, scope, mitigation and more. Ensuring that stakeholders across security, infrastructure, engineering and operations teams are informed and engaged is one of the chief responsibilities of the Security Incident Manager On Call. The
Security Incident Manager On Call should focus on providing high-level status updates without delving too deeply into the technical details of the incident, including:
Any time there is a service disruption for team members, the CMOC should post details regarding the disruption and related issue(s) in #whats-happening-at-gitlab, and cross-post in any related channels. It is important to identify if this is a production incident affecting gitlab.com or a service used by the organization.
In cases of high priority security notifications appropriate for the entire organization, the Internal Security Notification Dashboard should be used. When an update is made to this dashboard, notifications will be sent via slack and email to all GitLab team members.
In the cases of incidents that are on-going and require constant communication the
Security Engineer on Call will set up an incident response Slack channel. All security incident team members and extended POCs should be invited. If the nature of the incident allows, the Slack channel will be public to GitLab and a link to this channel will also be shared in
#security-department Slack channel to increase visibility.
|Group & Contacts||When to Engage||DRI to Engage||At what Cadence||In what Channel|
|Director of Security Operations||For S1 incidents immediately upon determination of the S1 severity rating||
||30 minute intervals (unless otherwise requested)||In incident response Slack channel|
|VP of Security||For S1 incidents immediately upon determination of the S1 severity rating||
||30 minute intervals (unless otherwise requested)||Slack direct message|
|Broader e-group||Immediately in cases of a data breach or an RCE with evidence of exploitation||
||30 minute intervals (unless otherwise requested)||
|Sr. Director of Corporate Marketing and Director of Corporate Communications||Immediately, if the incident has been publicly reported or if there is a regulatory requirement to make an announcement. In other cases, once the full impact and associated risk has been determined.||
||Continuous||In incident response Slack channel|
|Legal||If GitLab EE customers are impacted, or if the security incident includes a data breach including but not limited to:
Exposure of PII / Personal Data
||Continuous||Incident response Slack channel|
||As soon as we know we'll need to communicate with customers||
||Continuous||Incident response Slack channel|
External communications should happen as soon as possible after the scope and impact of the security incident is determined, using concise and clear language. The first external communications are directed to affected parties. Examples include: Affected customers and third parties (including JiHu), providers of products, services or organizations involved in the supply chain for system components as related to the incident. Regulatory authorities are contacted based on incident scope, regulatory and legal requirements.
Once it has been determined that external response is needed, the SIRT team should aim to develop, gain approval on a final customer communication and distribute and/or publish within 6 hours for an S1 severity vulnerability.
SIMOC/SecCMOCto track content development, reviews and approvals.
@heatherinto the issue for first review and consult on communications forms/channels (more details below).
@heatherin the related security incident external response issue and security incident slack channel for first round review and edits.
Security External Communicationsteam will engage the appropriate marketing teams for support (PR, Marketing Ops, Content teams) and begin creating the related issues for parallel marketing support and message deployment efforts.
Security External Communicationshas completed their review, the
SecCMOCshould route the communication doc to the
designated key approversas outlined below for their collaboration and review; being sure to note the time at which all approvals must be achieved (targeted deployment time is 6 hours, and includes 1-2 hours for information gather and communications drafting, and 1 hour of deployment time for blogs to be published and emails to be deployed). At this time, the
designated key approvershave 30 minutes to review and make any necessary edits to the doc. Reviews must be provided or a stand-in appointed within this 30 minute time limit.
designated key approversshould be tagged in slack for their final review and approval. This review and approval process should take no more than 30 minutes and approvers should mark their approval in the
approval matrixat the top of the working incident response comms google doc.
Security External Communicationswill work with the appropriate marketing and communications teams to deploy the communications to the appropriate external and internal audiences.
Security External Communicationswill ensure all final materials and associated deployment details are noted in the related communications issue and close the issue.
Our intent is to provide clear and accurate information to GitLab users as quickly as possible. In the case of an S1 (critical) vulnerability, the targeted turnaround time on approved and deployed communications is 6 hours. Because of this accelerated response window, our security team (SecCMOC or Incident Commander) will announce the targeted cut-off time for approvals at the beginning of the review and approvals process (6 hours from the start of communications drafts) and will move forward, if needed at that time, when at least 2 of the 4 required groups (Security, Legal, Customer Success and Investor Relations) have approved. See below for more information.
|Group||Blog||Customer Email||FAQs for Support Teams||Media Response||Social/Forum Response|
||Approver||Approver||Approver||Approver if quote attributed to Security, FYI otherwise||FYI Only|
||Approver||Approver||FYI Only||Approver||FYI Only|
|Support Manager on Call||FYI Only||FYI Only||FYI Only||FYI Only||FYI Only|
||FYI Only||FYI Only||FYI Only||DRI/Contributor||FYI Only|
||Approver||Approver||FYI Only||Approver||FYI Only|
Note: see this confidential issue for contact details
- For the "required" approvers, we need at least 2 designated back-up approvers.
VP Securityis not available, one of the
Security directorsmay approve, if not available move to
EVP of Engineering.
Senior Director of Legal, Privacy and Productcan approve. If not available, then move to
Legal Counsel, and then move to
VP Customer Successcan approve. If not available,
Snr Director, Customer Success Managers, then
Director of Customer Success, Public Sector.
Support Manager on callcan approve. If not available, move to a member of Support Senior Leadership, then to Support Managers if needed.
Director of Corp Commscan approve. If not available, move to
Manager, Public Relationsand then move to
Manager, Corp Commsif needed.
VP of Investor Relationscan approve. If not available, move to
Chief Financial Officer, and then move to
VP Finance and Business Technologyfor back-up approval.
Note: See this confidential issue for contact details.
designated key approvers
Designated key approver response time: 30 minutes
Per our crisis comms firm and GitLab's Corporate Incident Response plans, each approver has 30 minutes to respond and review communications (providing feedback or edits) before we engage back-ups. Once feedback from the approvers has been received, reviewed and consolidated, a final review from approvers will be requested and each approver has 30 minutes response time to provide their approval. If we are nearing the established deploy time for communications and a given teams' review and approval is not achieved within 30 minutes, we will move forward to finalization and deployment once we have approval from at least 2 of the 4 required groups (Security, Legal, Customer Success and Investor Relations). See below for more information.
designated key approverson an urgent basis
designated key approvers.
If 30 minutes has passed and the SecCMOC has not heard from a specific
designated key approver, we move to engage all of that individual's
designated back-up approvers in the incident-related slack channel and issue. If we are nearing the established deploy time for communications and a given teams' review and approval is not achieved within 30 minutes, we will move forward to finalization and deployment once we have approval from at least 2 of the 4 required groups (Security, Legal, Customer Success and Investor Relations).
The communications channels and forms that should be used in an incident or event can vary but should align with our need to be responsive to our customers and our transparency value, and be balanced with the potential risk and exposure to customers.
Commonly used forms and channels of communication:
Security external incident or event response template (this is an internal template) has links to templates that can be used (make a copy)to start various communications.
📝 Security incident communications runbooks are located here (internal only).
It is important to keep in mind, any time we are communicating externally, we need to advise our support, customer, social and community relations teams that we’ll be making external communication about an issue that affects customers and/or the community.
For this reason, each incident response (direct email, media statement, blog post, etc) should have accompanying:
For on-going, live site incidents on GitLab.com, updates are provided by the
SIMOC/CMOC through status.io to https://status.gitlab.com/ and the @GitLabStatus twitter handle.
Depending on scope, impact or risk associated with the incident, our Corporate Communications and Marketing team may determine that additional outreach is necessary. Any official statements about the security incident would be made by GitLab’s Director of Corporate Communications, VP of Corporate Marketing, CMO or VP of Security.
|Communications Channels||Purpose/Message||Additional Details|
|Incident Response Customer Email||Provides incident background, response, potential, impact, follow-up actions, and who to contact with questions.||Drafted by SIMOC/CMOC and reviewed by DRIs from Support, Legal, External Comms and Security. Sent from firstname.lastname@example.org. Should be in plain text with no link tracking. If an accompanying blog post is published, blog should be linked.|
|Mitigation and response blog post||Details the background, GitLab response and any action required by our customers.||Developed when it is determined that a longer, more-in-depth response is needed, Content for the blog post is provided by the
|GitLab Security Release Alert/Email||Indicates required action for customers and links to related mitigation and response blog.||Email sent to opt-in security notices distribution list. If a related blog post has been published, this email should include a link. Prepared and sent by
|Customer Frequently Asked Questions (FAQs)||List of early customer questions and responses, or probable questions and responses.||Created by
|Social media post||For distribution of related blog post, details our response to X issue.||
When appropriate, key stakeholders for contribution, review and approval should meet synchronously in a Zoom session to create and fine-tune customer communications (emails, FAQs, blog post, etc). Meeting synchronously in this case allows us to expedite the development of communications with key inputs from stakeholders in security, customer support and beyond and quickly move into the review stage. These zoom sessions are recorded and will be linked into the related security incident external response issue.
Exceptions to this procedure will be tracked as per the Information Security Policy Exception Management Process.
This sample issue template can used be to route, review and track approvals on the communications surrounding critical security incidents. As this is a sample, you would need to adjust/modify/tailor as necessary to suit your own organization's structure, processes and needs. Given that each organization is different and critical security incidents may vary, GitLab cannot guarantee any results with respect to your use of the template.