We enhance the security posture of our company, products, and client-facing services. The security team works cross-functionally inside and outside GitLab to meet these goals. The security team does not work directly on security-centric features of our platform—these are handled by the development teams. The specialty areas of our team reflect the major areas of information security.
If you identified an urgent security issue, if something feels wrong, or you need immediate assistance from the Security Team, you have two options available:
/security Hi security team, I have a concern! Please see the following URL ...command.
Both mechanisms will trigger the very same process, and as a result, the Security responder on-call will engage in the relevant issue within the appropriate SLA. If the SLA is breached, the Security manager on-call will be paged.
Note: The Security Team will not be mad at you if you page us for something that turns out to be nothing, but we will be mad if you don't page us for something that turns out to be something. IF UNSURE - PAGE US.
Security Automation specialists help us scale by creating tools that perform common tasks automatically. Examples include building automated security issue triage and management, proactive vulnerability scanning, and defining security metrics for executive review. Initiatives for this specialty also include:
Application Security specialists work closely with development, product security PMs, and third-party groups (including paid bug bounty programs) to ensure pre and post deployment assessments are completed. Initiatives for this specialty also include:
Security Operations specialists respond to incidents. This is often a fast-paced and stressful environment, where responding quickly and maintaining ones composure is critical. Initiatives for this specialty also include:
Abuse Operations specialists investigate malicious use of our systems. Initiatives for this specialty include:
Compliance specialists enables Sales by achieving standard as required by our customers. This includes SaaS, on-prem, and open source instances. Initiatives for this specialty also include:
Threat intelligence specialists research and provide information about specific threats to help us protect from the types of attacks that could cause the most damage. Initiatives for this specialty also include:
Strategic security specialists focus on holistic changes to policy, architecture, and processes to reduce entire categories of future security issues. Initiatives for this specialty also include:
Security research specialists conduct internal testing against GitLab assets, and against FOSS that is critical to GitLab products and operations. Initiatives for this specialty also include:
The Security team will collaborate with development and product management for security-related features in GitLab. The Secure team must not be mistaken with the Security Team.
We work closely with bounty programs, as well as security assessment and penetration testing firms to ensure external review of our security posture.
~metaand backend tasks, and catch all for anything not covered by other other projects.
gl-security/runbooksshould only be used for documenting specifics that would increase risk and/or have customer impact if publicly disclosed. anything security on-call or other operational activities.
GitLab.comenvironment, consider if it's possible to release when the
~securityissue becomes non-confidential.
#incident-managementand other producton team channels
#abuse*- Multiple channels for different notifications handled by the Security Department.
Security crosses many teams in the company, so you will find
~security labelled issues across all GitLab projects, especially:
When opening issues, please follow the Creating New Security Issues process for using labels and the confidential flag.
At a high level, the topic of security encompasses keeping GitLab.com safe, from the perspective of the application, the infrastructure, and the organization.
We track progress on tackling security related issues across the spectrum through an "always WIP Risk Assessment", and its associated (confidential) meta issue that is kept up to date to list the top 10 actions the team is working on.
The definitions, processes and checklists for security releases are described in the release/docs project.
The policies for backporting changes follow Security Releases for Gitlab EE.
The Security team needs to be able to communicate the priorities of security related issues to the Product, Development, and Infrastructure teams. Here's how the team can set priorities internally for subsequent communication (inspired in part by how the support team does this).
New security issue should follow these guidelines when being created on
confidentialif unsure whether issue a potential vulnerability or not. It is easier to make an issue that should have been public open than to remediate an issue that should have been confidential. Consider adding the
/confidentialquick action to a project issue template.
~securityat a minimum.
~feature proposalif appropriate
~customerif issue is a result of a customer report
~keep confidential. If possible avoid this by linking resources only available to GitLab employess, for example, the originating Zendesk ticket. Label the link with
(GitLab internal)for clarity.
If necesary, a sanitized issue may need to be created with more general discussion and examples appropriate for public disclosure prior to release.
For more immediate attention, mention
@gitlab-com/gl-security with a request in the issue and use
@sec-team in Slack.
The presence of
~security modifies the standard severity labels(
~S4) by additionally taking into account likelihood as described below, as well as any other mitigating or exacerbating factors. The priority of addressing
~security issues is also driven by impact, so in most cases, the priority label assigned by the security team will match the severity label. Exceptions must be noted in issue description or comments.
The intent of tying
~S/~P labels to remediation times is to measure and improve GitLab's response time to security issues to consistently meet or exceed industry standard timelines for responsible disclosure.
|Severity||Priority||Time to remediate|
| || ||As soon as possible|
| || ||Within 60 days|
| || ||Within 90 days|
~security issues, the security engineer assigns the
Due date, which is the target date of when fixes should be ready for release. This due date should account for the
Time to remediate times above, as well as monthly security releases on the 28th of each month. For example, suppose today is October 1st, and a new
~security issue is opened. It must be addressed in a security release within 60 days, which is November 30th. So therefore, it must catch the November 28th security release. Furthermore, the Security Release Process deadlines say that it should the code fix should be ready by November 23rd. So the due date in this example should be November 23rd.
Note that some
~security issues may not need to be part of a code release, such as an infrastructure change. In that case, the due date will not need to account for monthly security release dates.
On occasion, the due date of such an issue may need to be changed if the security team needs to move up or delay a monthly security release date to accomodate for urgent problems that arise.
~security issues do not have a due date since they should be fixed as soon as possible, and a security release made available as soon as possible to accomodate it.
Security issues which are not at least
~S3, are most likely
~feature proposals that should be triaged and prioritized by a product manager. This includes suggestions without a well-defined path for implementation or requiring complex changes to the application or architecture to address.
~S4/~P4 may be used for
~feature proposals as assigned by a product manager and do not need assignment to a release.
The security engineer will add team labels (
~Deploy, etc.) and any additional labels as appropriate to the issue. The product manager and engineering team lead should be @ mentioned and followed up with as necessary as noted below for different severity levels.
The product manager will assign a
Milestone to communicate when work will be assigned to engineers. The
Due date and labels should not be changed if not met to provide accurate metrics on
Note that issues are not scheduled for a particular release unless the team leads add them to a release milestone and they are assigned to a developer.
Issues with an
S2 rating should be immediately brought to the attention of the relevant engineering team leads and product managers by tagging them in the issue and/or escalating via chat and email if they are unresponsive.
Issues with an
S1 rating have priority over all other issues and should be considered for a critical security release.
Issues with an
S2 rating should be scheduled for the next scheduled security release, which may be days or weeks ahead depending on severity and other issues that are waiting for patches. An
S2 rating is not a guarantee that a patch will be ready prior to the next security release, but that should be the goal.
Issues with an
S3 rating have a lower sense of urgency and are assigned a target of the next minor version. If a low-risk or low-impact vulnerability is reported that would normally be rated
S3 but the reporter has provided a 30 day time window (or less) for disclosure the issue may be escalated to ensure that it is patched before disclosure.
Centralized access management is key to ensuring that the correct GitLabbers have access to the correct data and systems and at the correct level. GitLab access controls are guided by the principle of least privilege and need-to-know. These controls apply to information and information processing systems at the application and operating system layers, including networks and network services.
The access request project is used to request and track the following access-related activities:
Usage guidelines for each of the access templates is outlined in the project's README file
Access Control Procedures
All access requests must be approved by the team member's manager.
Requests for access to Infrastructure assets (servers and databases) require a second layer of approval from Infrastructure Management.
If admin-level access is being requested, the request must be approved by the Gitlab Security Team.
In the case of a separation from the company, all access will be deprovisioned within 3 business days from the date on which the offboarding request is submitted unless otherwise specified.
Individual access removal requests will be processed within the SLA requested. If no SLA is noted, access will be deprovisioned within 3 business days of the submission of the issue.
If access removal needs to occur immediately, please follow the panic email procedures, which will alert the Security Team on-call.
GitLab's access controls include the following control activities:
For systems built (or significantly modified) by functional groups that house customer and other sensitive data, the Security Team should perform applicable application security reviews to ensure the systems are hardened.
The current process for requesting an internal application security review is:
As part of the app sec review process, a security issue is created to track the progress of the review. In those issues, labels and milestones can be added. It's important to note here that application security reviews are not a one-and-done, but can be ongoing as the application under review evolves.
The security team plays a large role in defining procedures for defending against and dealing with spam. Common targets for spam are public snippets, projects, issues, merge requests, and comments. Advanced techniques for dealing with these types of spam are detailed in the Spam Fighting runbook.
For any actions taken on an account:
The purpose of adding Admin Notes allow us to better assist the Support Team and Production if there are any questions around changes made to an account by the Security Team.
The Security Team plays a big role in defining the procedures and reviewing Digital Millenium Copyright Act (DMCA) requests. All DMCA requests need to be vetted by Legal first before any further steps are taken to proceed with the take down of reported content. Reported content that has been sucessfully vetted by Legal must be referred to the Abuse Team before any action is taken.
Abuse works in conjuction with Support and Legal referencing the DMCA Removal Workflow
GitLab has an "always WIP" Risk Assessment and all team members are encouraged to participate. The Risk Assessment consists of a list of all risks or threats to the GitLab infrastructure and GitLab as a company, their likelihood of occurring, the impact should they occur, and what actions can be taken to prevent these risks from damaging the company or mitigate the damage should they be realized.
The risk assessment is stored as a Google Sheet (search Google Drive for "Risk Assessment"; make sure you are searching for spreadsheets shared with GitLab.com) and is available to all team members. It should not be shared with people outside of the company without permission.
The format of the Risk Assessment may seem intimidating at first. If you do not know what values to use for risk ratings, impact ratings, likelihoods or any other value leave them blank and reach out to the Security Team to help you determine appropriate values. It is more important to have all risks documented than it is to have all values completed when adding new risks. Guidelines and instructions for how to add a risk and how to calculate each rating or score are included on the "Instructions" tab.
GitLab receives vulnerability reports by various pathways, including:
For any reported vulnerability:
devor in other non-public ways even if there is a reason to believe that the vulnerability is already out in the public domain (e.g. the original report was made in a public issue that was later made confidential).
GitLab utilizes HackerOne for its bug bounty program. Security researchers can report vulnerabilities in GitLab applications or the GitLab infrastructure via the HackerOne website. Team members authorized to respond to HackerOne reports use procedures outlined here.
Add commentusing the
Week to respondtemplate.
~feature proposalas defined above and would not need to be made confidential or scheduled for remediation. An issue can be created, or requested that the reporter creates one if desired, but the report can be closed as "Informational".
~HackerOnelabel to these issues, for later reporting and tracking.
Traiged-Escalated to engineeringCommon response as a template.
If a report is unclear, or the reviewer has any questions about the validity of the finding or how it can be exploited, now is the time to ask. Move the report to the "Needs More Info" state until the researcher has provided all the information necessary to determine the validity and impact of the finding. Use your best judgement to determine whether it makes sense to open a confidential issue anyway, noting in it that you are seeking more information from the reporter. When in doubt, err on the side of opening the issue.
One the report has been clarified, follow the "regular flow" described above.
If a report violates the rules of GitLab's bug bounty program use good judgement in deciding how to proceed. For instance, if a researcher has tested a vulnerability against GitLab production systems (a violation), but the vulnerability has not placed GitLab user data at risk, notify them that they have violated the terms of the bounty program but you are still taking the report seriously and will treat it normally. If the researcher has acted in a dangerous or malicious way, inform them that they have violated the terms of the bug bounty program and will not receive credit. Then continue with the "regular flow" as you normally would.
If the report is invalid (in your determination) or does not pose a security risk to GitLab or GitLab users it can be closed without opening an issue on GitLab.com. When this happens inform the researcher why it is not a vulnerability and close the issue as "Informational". HackerOne offers the option to close an issue as "Not Applicable" or "Spam". Both of these categories result in damage to the researcher's reputation and should only be used in obvious cases of abuse.
When a patch has been developed, tested, approved, merged into the security branch, and a new security release is being prepared it is time to inform the researcher via HackerOne. Post a comment on the HackerOne issue to all parties informing them that a patch is ready and will be included with the next security release. Provide release dates, if available, but try not to promise a release on a specific date if you are unsure.
This is also a good time to ask if they would like public credit in our release blog post and on our vulnerability acknowledgements page for the finding. We will link their name or alias to their HackerOne profile, Twitter handle, Facebook profile, company website, or URL of their choosing. Also ask if they would like the HackerOne report to be made public upon release. It is always preferable to publicly disclose reports unless the researcher has an objection.
We use CVE IDs to uniquely identify and publicly define vulnerabilities in our products. Since we publicly disclose all security vulnerabilities 30 days after a patch is released, CVE IDs must be obtained for each vulnerability to be fixed. The earlier obtained the better, and it should be requested either during or immediately after a fix is prepared.
We currently request CVEs either through the HackerOne team or directly through MITRE's webform. Keep in mind that some of our security releases contain security related enhancements which may not have an associated CWE or vulnerability. These particular issues are not required to obtain a CVE since there's no associated vulnerability.
On the day of the security release several things happen in order:
Once all of these things have happened notify the HackerOne researcher that the vulnerability and patch are now public. The GitLab issue should be closed and the HackerOne report should be closed as "Resolved". Public disclosure should be requested if they have not objected to doing so. Any sensitive information contained in the HackerOne report should be sanitized before disclosure.
GitLab awards swag codes for free GitLab swag to any reports that result in a security patch. Limit: 1 per reporter. When a report is closed, ask the reporter if they would like a swag code for free GitLab clothing or accessories. Swag codes are available by request from the marketing team.
Some customers, to keep up with regulations that impact their business, need to understand the security implications of installing any software - including software like GitLab.
The current process for responding to customer requests is:
SA Backlogfor the completion of that document
GitLab maintains a custom vulnerability scanner that is used to regularly scan all GitLab assets for common vulnerabilities as well as previously patched GitLab vulnerabilities and to ensure that no GitLab security-sensitive services are accidentally exposed.
Details on this scanner and how it is configured are available to all team members in a Google Doc entitled "Vulnerability Scanner Config".
The packages we ship are signed with GPG keys, as described in the omnibus documentation. The process around how to make and store the key pair in a secure manner is described in the runbooks. Those runbooks also point out that the management of the keys is handled by the Security team and not the Build team. For more details that are specific to key locations and access at GitLab, find the internal google doc titled "Package Signing Keys at GitLab" on Google Drive.