Security

1. You are here:
2. Handbook
3. Engineering
4. Security

On this page

• Security Vision
• Engaging the Security On-Call
• External Contact Information
• Security Department
  • Security Automation
  • Application Security
  • Security Operations
  • Abuse Operations
  • Compliance
  • Threat Intelligence
  • Strategic Security
  • Security Research
  • Security External Communications
  • Red Team
• Security Team Collaborators
  • Secure Team
  • External Security Firms
• Resources
  • Information Security Policies
  • GitLab.com Groups and Projects
  • Slack Channels
  • Other Frequently Used GitLab.com Projects
  • Other Resources for GitLab Team Members
  • How to track OKR related work scheduled by quarter
• Security Releases
• Issue Triage
  • Creating New Security Issues
  • Severity and Priority Labels on ~security Issues
  • Due date on ~security Issues
    • ~feature
    • ~"security request"
    • Transferring from Security to Engineering
    • Security issue becoming irrelevant due to unrelated code changes
• Access Management Process
  • Access Control Policy and Procedures
  • Bulk Access Requests
  • Access Requests and Onboarding
  • Deprovisioning
  • Access Reviews
  • Access Control Process Exceptions
  • Baseline Role-Based Entitlements Access Runbooks & Issue Templates
  • Access Control Procedure Activities
  • Account Naming Conventions
  • Automated Group Membership Reports for Managers
• Internal Application Security Reviews
  • When to request a security review?
  • How to request a security review?
    • Individual merge requests
    • Larger scale initiatives
  • Is security approval required to progress?
  • What should I provide when requesting a security review?
  • What does the security process look like?
  • My changes have been reviewed by security, so is my project now secure?
• Fighting Spam
• DMCA Requests
  • For DMCA requests the Abuse Team will follow the below process
• Vulnerability Reports and HackerOne
  • HackerOne Process
    • Triage Rotation
    • Working the Queue
    • If a Report is Unclear
    • If a Report Violates the Rules
    • If the Report is Invalid
  • When a Patch is Ready
    • CVE IDs
  • On Release Day
  • Swag for Reports
  • Handling Disruptive Researcher Activity
• External Code Contributions
• Security Questionnaires for Customers
  • Process
• Vulnerability Scanning
• Package Signing

Security Vision

We enhance the security posture of our company, products, and client-facing services. The security team works cross-functionally inside and outside GitLab to meet these goals. The security team does not work directly on security-centric features of our platform—these are handled by the development teams. The specialty areas of our team reflect the major areas of information security.

Engaging the Security On-Call

If you identified an urgent security issue, if something feels wrong, or you need immediate assistance from the Security Team, you have two options available:

• Slack: use the /security Hi security team, I have a concern! Please see the following URL ... command.
• Email: send an email with a brief description of the issue to page-security@gitlab.com.

Please be aware that the Security Team can only be paged internally. If you are an external party, please proceed to Vulnerability Reports and HackerOne section of this page.

Both mechanisms will trigger the very same process, and as a result, the Security responder on-call will engage in the relevant issue within the appropriate SLA. If the SLA is breached, the Security manager on-call will be paged.

Note: The Security Team will never be upset if you page us for something that turns out to be nothing, but we will be sad if you don't page us and it turns out to be something. IF UNSURE - PAGE US.

External Contact Information

The security team can be contacted at security@gitlab.com. External researchers or other interested parties should refer to our Responsible Disclosure Policy for more information about reporting vulnerabilities. The security@gitlab.com email address also forwards to a ZenDesk queue that is monitored by the security team.

For security team members, the private PGP key is available in the Security 1Password vault. Refer to PGP process for usage.

Security Department

Security Automation

Security Automation specialists help us scale by creating tools that perform common tasks automatically. Examples include building automated security issue triage and management, proactive vulnerability scanning, and defining security metrics for executive review. Initiatives for this specialty also include:

• Assist other security specialty teams in their automation efforts
• Assess security tools and integrate tools as needed
• Define and own metrics and KPIs to determine the effectiveness of security programs
• Define, implement, and monitor security measures to protect GitLab.com and company assets

Application Security

Application Security specialists work closely with development, product security PMs, and third-party groups (including paid bug bounty programs) to ensure pre and post deployment assessments are completed. Initiatives for this specialty also include:

• Pre and post deployment security assessments/tests
• Capture of flaws in software environment configuration
• Malicious code detection
• Patch/upgrade
• IP filtering
• Lock down executables
• Monitoring of programs at runtime to enforce the software use policy
• Developing and implementing a Secure Software Development Lifecycle (S-SDLC)
• Training and coaching developers on current security best practices

Security Operations

Security Operations specialists respond to incidents. This is often a fast-paced and stressful environment, where responding quickly and maintaining ones composure is critical. Initiatives for this specialty also include:

• Focus on detection and incident response, as well as implement preventative mechanisms
• Coordinate company-wide responses to security incidents
• Incorporate current security trends, advisories, publications, and academic research
• Engineer CND technologies to monitor and analyze e.g.
• IDSes
• Data collection tools
• Conduct vulnerability management
  • Identify and mitigate complex security vulnerabilities before an attacker exploits them
  • Determine level of effort required to compromise sensitive data
  • Triage and manage vulnerabilities identified through scanning

Security Operations can be contacted on slack via our handle @sec-ops-team or in a Gitlab issue using @gitlab-com/gl-security/secops. If your request requires immediate attention please review the steps for engaging the security on-call.

Abuse Operations

Abuse Operations specialists investigate malicious use of our systems. Initiatives for this specialty include:

• Three main types of abuse
  • Continuous Integration
  • Copyright violations
  • Port scanning
• Also investigate reports such as spam email, DoS, etc.
• Take action on abusive/non-responsive customers
• Verify proper classification of incoming abuse reports
• Execute messaging to customers on best practices
• Escalate to other stakeholders while continuing to monitor
• Monitor logs and queues for trends
• Research and prevent trending abuse methodologies

Compliance

Compliance enables Sales by achieving standards as required by our customers and help to verify that the outcomes the security team is trying to achieve are actually being met. This includes SaaS, on-prem, and open source instances. Initiatives for this specialty also include:

• Develop roadmap based on customer needs e.g.
  • GDPR
  • SOC 2
  • FIPS 140-2
• Align other security specialist activities with the compliance roadmap
• Develop relationships with key government personnel and policy makers
• Assist work of internal and external auditors or advisors as needed

Please refer to GitLab's security controls for a detailed list of all compliance controls organized by control family.

Threat Intelligence

Threat intelligence specialists research and provide information about specific threats to help us protect from the types of attacks that could cause the most damage. Initiatives for this specialty also include:

• Collect and analyze threat intelligence reports covering new threats, vulnerabilities, products, and research
• Author threat intelligence reports, driven by our security operations team's own incidents, analysis, and adversary engagements
• Evolve monitoring operations by extracting data from threat intelligence and create new content, signatures, and understanding of adversary TTPs
• Analyze event feeds and collected malware over long term to trend and correlate

Strategic Security

Strategic security specialists focus on holistic changes to policy, architecture, and processes to reduce entire categories of future security issues. Initiatives for this specialty also include:

• Cluster related historical security issues and examine them as a larger set
• Identify and generate trends associated with each set
• Propose actionable changes to GitLab architecture, processes, and infrastructure to mitigate future issues within each set
• Generate metrics which measure the effectiveness of each mitigation implemented

Security Research

Security research specialists conduct internal testing against GitLab assets, and against FOSS that is critical to GitLab products and operations. Initiatives for this specialty also include:

• Conduct vulnerability research against all GitLab and GitLab.com assets
• Research FOSS tools that are integrated with GitLab
• Develop proof-of-concept code to be included in security findings
• Report findings to tool developers and track mitigation process
• Follow responsible disclosure policies for community disclosure
• Author blog posts on vulnerabilities discovered

Security External Communications

The External Communications Team leads customer advocacy, engagement and communications in support of GitLab Security Team programs. Initiatives for this specialty include:

• Increase engagement with the hacker community, including our public bug bounty program.
• Build and manage a Security blogging program.
• Develop social media content and campaigns, in collaboration with GitLab social media manager.
• Manage security alert email notifications.
• Collaborate with corporate marketing, PR, Community Advocates and Technical Evangelism teams to help identify opportunities for the Security Team to increase industry recognition and thought leadership position.

Red Team

Red team extend the objectives of penetration testing by examining the security posture of organizations and their ability to implement effective cyber defenses. Penetration testing is a specialized type of assessment conducted on information systems or individual system components to identify vulnerabilities that could be exploited by adversaries. Such testing can be used to either validate vulnerabilities or determine the degree of resistance organizational information systems have to adversaries within a set of specified constraints (e.g., time, resources, and/or skills). Red team exercises provide more comprehensive assessments that reflect real-world conditions over penetration testing. The exercises can further be used to improve security awareness and training and to assess levels of security control effectiveness. GitLab utilizes NIST 800-53 Revision 4 security control CA-8 to define the Red Team and their mission. The control can be found on NIST.gov.

• Assess the defensive capabilities of GitLab Defend and identify areas of improvement to help guide the product's direction.
• Assess the efficacy of the zero trust implementation on GitLab.com.
• Perform exercises with SecOps to collaboratively and rapidly iterate on improving GitLab's security posture. These exercises will be referred to as purple team exercises merging blue (secops) and red teams together.
• Perform exercises to reflect simulated adversarial attempts to compromise organizational mission/business functions and provide a comprehensive assessment of the security state of information systems and organizations.
• Simulate adversarial attempts to compromise organizational missions/business functions and the information systems that support those missions/functions may include technology-focused attacks (e.g., interactions with hardware, software, or firmware components and/or mission/business processes) and social engineering-based attacks (e.g., interactions via email, telephone, shoulder surfing, or personal conversations).

Security Team Collaborators

Secure Team

The Security team will collaborate with development and product management for security-related features in GitLab. The Secure team must not be mistaken with the Security Team.

• Develop real-world security use cases
• Ideation of compelling security features and products
• Test and provide product feedback

External Security Firms

We work closely with bounty programs, as well as security assessment and penetration testing firms to ensure external review of our security posture.

• One security assessment firm is rotated periodically to gain new perspectives
• One security assessment firm is engaged every quarter to build expertise in GitLab
• We maintain public and private bug bounty programs via HackerOne
  • Private program paid bounties will increase over time and at some point we want to do public bounties

Resources

Information Security Policies

• GitLab Internal Acceptable Use Policy

• GitLab Password Policy

• Access Management Policy

• Data Classification Policy

GitLab.com Groups and Projects

• @gitlab-com/gl-security is the primary group for @'mentioning the entire Security Department.
• @gitlab-com/gl-security/security-managers is the primary group for @'mentioning all managers in the Security Department.
• public (!) Security Engineering for Security Department initiatives, ~meta and backend tasks, and catch all for anything not covered by other projects.
• @gitlab-com/gl-security/compliance is the primary group for @'mentioning the Compliance team.
• @gitlab-com/gl-security/appsec is the primary group for @'mentioning the Application Security team.
• @gitlab-com/gl-security/secops is the primary group for @'mentioning the Security Operations team.
• @gitlab-com/gl-security/automation is the primary group for @'mentioning the Security Automation team.
• @gitlab-com/gl-security/abuse-team is the primary group for @'mentioning the Abuse Operations team.
• Security Operations (private) for
• security runbooks (private) - NOTE: The production runbooks should be the first location considered for any process or documentation. gl-security/runbooks should only be used for documenting specifics that would increase risk and/or have customer impact if publicly disclosed.
• Incident-Tools (private) for working scripts and other code during or while remediating an incident. If the tool is applicable outside of the GitLab.com environment, consider if it's possible to release when the ~security issue becomes non-confidential. This group can also be used for private demonstration projects for security issues.
• security-tools (mostly private) contains some operational tools used by the security team. Contents and/or configurations require that most of these projects remain private.

Slack Channels

• #security; Can be used for reporting when unsure of where to go, also used for general questions and discussion that often comes from the great questions people ask.
• #incident-management and other infrastructure team channels
• #security-alert-manual - New reports for the security team from various intake sources, including ZenDesk and new HackerOne reports.
• #hackerone-feed - Feed of most activity from our HackerOne program.
• Other #security-alert-* and #abuse* - Multiple channels for different notifications handled by the Security Department.
• Use the @sec-ops-team mention in any Slack channel to tag the members of the Security Operations team.
• Use the @sec-compliance-team mention in any Slack channel to tag the members of the Security Compliance team.
• Use the @appsec-team mention in any Slack channel to tag the members of the Application Security team.
• Use the @abuse-team mention in any Slack channel to tag the members of the Abuse Operations team.

Other Frequently Used GitLab.com Projects

Security crosses many teams in the company, so you will find ~security labelled issues across all GitLab projects, especially:

• gitlab-ce
• gitlab-ee
• infrastructure
• production

When opening issues, please follow the Creating New Security Issues process for using labels and the confidential flag.

Other Resources for GitLab Team Members

• Security Best Practices, using 1Password and similar tools, are documented on their own security best practices page.
• GitLab.com and GitLab Hosted data breach notification policy.
• GitLab Internal Acceptable Use Policy.
• For GitLab.com, we have developed a Google Cloud Platform (GCP) Security Guidelines Policy document, which outlines recommended best practices, and is enforced through our security automation initiatives.

• GitLab Security Tanuki for use on security release blogs, social media and security related swag as appropriate: Web-RGB

  Print-CMYK and one exclusively for stickers.

How to track OKR related work scheduled by quarter

The current board for tracking current Security Department Work is Security Management FY19Q2 This board aggregrates work from the many subgroups and projects under gitlab-com/gl-security used by the various subteams to track their work and projects. High level issues and open issues that map directly to an OKR should be added to this board to provide visibility into the current work of the security team.

When creating an issue that maps to an OKR, apply the following group level labels so that it appears in the board:

• The label for the assigned (usually the current) quarter. For example: ~FY19Q2
• The scoped Security Management:: label for the appropriate team. For example: ~Security Management::Security Automation Team

These labels can also be applied to MRs related to the work.

Security Releases

The definitions, processes and checklists for security releases are described in the release/docs project.

The policies for backporting changes follow Security Releases for Gitlab EE.

For critical security releases, refer to Critical Security Releases in the handbook for a high level description of communication, and the critical release checklist in release/docs.

Issue Triage

The Security team needs to be able to communicate the priorities of security related issues to the Product, Development, and Infrastructure teams. Here's how the team can set priorities internally for subsequent communication (inspired in part by how the support team does this).

Creating New Security Issues

New security issue should follow these guidelines when being created on GitLab.com:

• Create new issues as confidential if unsure whether issue a potential vulnerability or not. It is easier to make an issue that should have been public open than to remediate an issue that should have been confidential. Consider adding the /confidential quick action to a project issue template.
• Always label as ~security at a minimum.
• Add any additional labels you know apply. Additional labels will be applied by the security team and other engineering personnel, but it will help with the triage process:
  • ~bug or ~feature if appropriate
  • Team or devops lifecycle labels
  • ~customer if issue is a result of a customer report
  • ~internal customer should be added by team members when the issue impacts GitLab operations.
• Issues that contain customer specific data, such as private repository contents, should be assigned ~keep confidential. If possible avoid this by linking resources only available to GitLab employees, for example, the originating ZenDesk ticket. Label the link with (GitLab internal) for clarity.

Occasionally, data that should remain confidential, such as the private project contents of a user that reported an issue, may get included in an issue. If necessary, a sanitized issue may need to be created with more general discussion and examples appropriate for public disclosure prior to release.

For review by the Application Security team, @ mention @gitlab-com/gl-security/appsec.

For more immediate attention, refer to Engaging security on-call.

Severity and Priority Labels on ~security Issues

The presence of ~security modifies the standard severity labels(~S1, ~S2, ~S3, ~S4) by additionally taking into account likelihood as described below, as well as any other mitigating or exacerbating factors. The priority of addressing ~security issues is also driven by impact, so in most cases, the priority label assigned by the security team will match the severity label. Exceptions must be noted in issue description or comments.

The intent of tying ~S/~P labels to remediation times is to measure and improve GitLab's response time to security issues to consistently meet or exceed industry standard timelines for responsible disclosure.

Due date on ~security Issues

For ~S2 and ~S3 ~security issues, the security engineer assigns the Due date, which is the target date of when fixes should be ready for release. This due date should account for the Time to remediate times above, as well as monthly security releases on the 28th of each month. For example, suppose today is October 1st, and a new S2 ~security issue is opened. It must be addressed in a security release within 60 days, which is November 30th. So therefore, it must catch the November 28th security release. Furthermore, the Security Release Process deadlines say that it should the code fix should be ready by November 23rd. So the due date in this example should be November 23rd.

Note that some ~security issues may not need to be part of a code release, such as an infrastructure change. In that case, the due date will not need to account for monthly security release dates.

On occasion, the due date of such an issue may need to be changed if the security team needs to move up or delay a monthly security release date to accommodate for urgent problems that arise.

~S1 ~security issues do not have a due date since they should be fixed as soon as possible, and a security release made available as soon as possible to accommodate it.

~feature

Security issues which would be triaged as ~S4, are most likely ~features that should be triaged and prioritized by a product manager. This includes suggestions without a well-defined path for implementation or requiring complex changes to the application or architecture to address. ~S4/~P4 may be used for ~features as assigned by a product manager and do not need assignment to a release.

~"security request"

The security team may also apply ~internal customer and ~security request to issue as an indication that the feature is being requested by the security team to meet additional customer requirements, compliance or operational needs in support of GitLab.com.

Transferring from Security to Engineering

The security engineer must:

• Add team label (~Create, ~Deploy, etc.)
• Add stage label
• Any additional labels, such as ~merge request.
• Mention the product manager for scheduling, such as @pm for scheduling.
• The engineering team lead should be @ mentioned and followed up with when necessary as noted below for different severity levels.

The product manager will assign a Milestone to communicate when work will be assigned to engineers. The Due date field, severity label, and priority label should not be changed by PMs, as these labels are intended to provide accurate metrics on ~security issues, and are assigned by the Security Team.

Note that issues are not scheduled for a particular release unless the team leads add them to a release milestone and they are assigned to a developer.

Issues with an S1 or S2 rating should be immediately brought to the attention of the relevant engineering team leads and product managers by tagging them in the issue and/or escalating via chat and email if they are unresponsive.

Issues with an S1 rating have priority over all other issues and should be considered for a critical security release.

Issues with an S2 rating should be scheduled for the next scheduled security release, which may be days or weeks ahead depending on severity and other issues that are waiting for patches. An S2 rating is not a guarantee that a patch will be ready prior to the next security release, but that should be the goal.

Issues with an S3 rating have a lower sense of urgency and are assigned a target of the next minor version. If a low-risk or low-impact vulnerability is reported that would normally be rated S3 but the reporter has provided a 30 day time window (or less) for disclosure the issue may be escalated to ensure that it is patched before disclosure.

Security issue becoming irrelevant due to unrelated code changes

It is possible that a ~security issue becomes irrelevant after it was initially triaged, but before a patch was implemented. For example, the vulnerable functionality was removed or significantly changed resulting in the vulnerability not being present anymore.

If an engineer notices that an issue has become irrelevant, he should @-mention the person that triaged the issue to confirm that the vulnerability is not present anymore. Note that it might still be necessary to backport a patch to previous releases according to our maintenance policy. In case no backports are necessary, the issue can be closed.

Access Management Process

Centralized access management is key to ensuring that the correct GitLab team-members have access to the correct data and systems and at the correct level. GitLab access controls are guided by the principle of least privilege and need-to-know. These controls apply to information and information processing systems at the application and operating system layers, including networks and network services.

The access request project is used to request and track the following access-related activities:

1. New Access Requests
2. Access Removal Requests
3. Access Reviews
4. New Service Account Requests

Usage guidelines for each of the access templates is outlined in the project's README file

These templates should be used during the onboarding process and throughout the employment tenure of a GitLabber. Access required as part of the team member's onboarding should be requested using the New Access Requests or if applicable, one of the available Baseline Role-based Entitlements templates.

Access Control Policy and Procedures

• All new access or permissioning change requests require a New Access Request.

• All access requests must be approved by the team member's manager with the exception of:

  • ARs for G-Suite email distribution lists for internal GitLab team members
  • ARs for Slack groups for internal GitLab team members

  Please note that ARs for access to internal systems for "external to GitLab individuals" (eg. customers, prospects) require managerial approval. This includes access to G-Suite security groups also require managerial approval.

• Requests for access to Infrastructure assets (servers and databases) require a second layer of approval from Infrastructure Management.

• All requests for new service accounts require a New Service Account Request

• All requests for new service accounts must be approved by a member of Infrastructure Management.

• In regard to support during or prior to provisioning, please do not tag the Security Operations team in the AR issue; to ask Security for help with AR assignments, please use the #security-itops channel.

Bulk Access Requests

• Bulk access requests are those that are for multiple GitLab team-members. Bulk requests can be submitted for the following three types of requests:

  • Email aliases/forwarding in GSuite.
  • The adding of team members to Slack groups.
  • All GitLab team-members listed on the Access Request have the same manager and the same level of access is being requested.

  Please note that the above use cases do not apply to ADMIN-level access, which needs to be submitted using the one issue per GitLab team-member rule.

• If admin-level access is being requested, the request must be approved by the team member's manager and Infrastructure Management if applicable.

Access Requests and Onboarding

During the onboarding process, the manager should determine which email and slack groups the new team member should be added to. Also determine if new team member will need access to the dev server, which is used by engineers to prepare fixes for security issues and also allows for access to version.gitlab.com and license.gitlab.com. If so, request the creation of a new dev.GitLab.org account with the same username the team member has on gitlab.com and an invitation to the gitlab group as a Developer. Fill out one access request for both the groups and Dev account if needed.

Deprovisioning

• In the case of a separation from the company, all access will be deprovisioned within 3 business days from the date on which the offboarding request is submitted unless otherwise specified.

• All attempts will be made for individual access removal requests to be processed within the SLA requested. If no SLA is noted, access will be deprovisioned within 3 business days of the submission of the issue.

• If access removal needs to occur immediately, please follow the panic email procedures, which will alert the Security Team on-call.

Access Reviews

• Access reviews will be formally documented using the Access Reviews template.

• As part of an access review, existing access may be modified or revoked. New access (not modification of existing access) requires the submission of a New Access Request.

Access Control Process Exceptions

• An access request is not required for Google Drive folders or files.

• Managerial approval is not required when requesting access to:

  • ARs for G-Suite email distribution lists for internal GitLab team members
  • ARs for Slack groups for internal GitLab team members

• Bulk requests can be submitted for the following three types of requests:

  • Email aliases/forwarding in GSuite
  • The adding of team members to Slack groups.
  • All GitLabbers listed on the Access Request have the same manager and the same level of access is being requested.

Baseline Role-Based Entitlements Access Runbooks & Issue Templates

• For certain roles, baseline entitlement access runbooks & templates that outline the baseline access configurations for the role have been established.

• The Baseline Role-based Entitlements directory contains all the approved role-based entitlement runbooks.

• The runbook is used to document the configuration and approvals for the baseline entitlements. Any changes to these approved configurations require the approval of Engineering and Security Management.

• Runbooks & Baseline Access Request Templates have been established for the following roles:

  • Site Reliability Engineer - SRE

• Security Engineer, Operations - SecOps

• Security Engineer, Automation

• FrontEnd Engineer

• Backend Engineer

When submitting a new access-request issue for either of these roles, please chose the template corresponding with the role of the person for which you are submitting the access request.

Access Control Procedure Activities

GitLab's access controls include the following control activities:

1. user registration and de-registration
2. user access provisioning
3. removal of adjustment of user access rights
4. management of privileged access rights
5. management and use of secret authentication information
6. review and recertification of user access rights
7. secure log-on procedures
8. management of passwords and tokens
9. access to privileged utility programs
10. access to program source code

Account Naming Conventions

• GitLab.com Administrator Account Naming Convention:
• Use user's GitLab G-Suite email +admin@gitlab.com as the email address: username+ADMIN@gitlab.com
• GitLab.com admin account name should the user's normal account with -admin appended: username-admin

• The admin account "Full name" should include text to indicate it is an admin account: First Last (Admin)

• Temporary Contractor Account Naming Convention: username-CTR@gitlab.com

Automated Group Membership Reports for Managers

If you would like to check whether or not a team-member is a member of a Slack or a G-Suite group, you can view the following automated group membership reports:

G-Suite Group Membership Reports

Slack Group Membership Reports

Internal Application Security Reviews

For systems built (or significantly modified) by functional groups that house customer and other sensitive data, the Security Team should perform applicable application security reviews to ensure the systems are hardened. Security reviews aim to help reduce vulnerabilities and to create a more secure product.

When to request a security review?

1. If your changes are processing, storing, or transferring any kind of RED or ORANGE data, it should be reviewed by the application security team.
2. If your changes involve implementing, utilizing, or is otherwise related to any type of authentication, authorization, or session handling mechanism, it should be reviewed by the application security team.
3. If your changes have a goal which requires a cryptographic function such as: confidentiality, integrity, authentication, or non-repudiation, it should be reviewed by the application security team.

How to request a security review?

There are two ways to request a security review depending on how significant the changes are. It is divided between individual merge requests and larger scale initiatives.

Individual merge requests

Loop in the application security team by /cc @gitlab\-com/gl\-security/appsec in your merge request.

These reviews are intended to be faster, more lightweight, and have a lower barrier of entry.

Larger scale initiatives

To get started, create an issue in the security tracker, add the app sec review label, and submit a triage questionnaire form. The complete process can be found at here.

Some use cases of this are for epics, milestones, reviewing for a common security weakness in the entire codebase, or larger features.

Is security approval required to progress?

No, code changes do not require security approval to progress. Non-blocking reviews enables the freedom for our code to keep shipping fast, and it closer aligns with our values of iteration and efficiency. They operate more as guardrails instead of a gate.

What should I provide when requesting a security review?

To help speed up a review, it's recommended to provide any or all of the following:

• The background and context of the changes being made.
• Any documentation or diagrams which help provide a clear understanding it's purpose and use cases.
• The type of data it's processing or storing.
• The security requirements for the data.
• Your security concerns and a worst case scenario that could happen.
• A test environment.

What does the security process look like?

The current process for larger scale internal application security reviews be found here

My changes have been reviewed by security, so is my project now secure?

Security reviews are not proof or certification that the code changes are secure. They are best effort, and additional vulnerabilities may exist after a review.

It's important to note here that application security reviews are not a one-and-done, but can be ongoing as the application under review evolves.

Fighting Spam

The security team plays a large role in defining procedures for defending against and dealing with spam. Common targets for spam are public snippets, projects, issues, merge requests, and comments. Advanced techniques for dealing with these types of spam are detailed in the Spam Fighting runbook.

For any actions taken on an account:

• Verify that the report has been fully resolved.
• Using Admin notes, add clear and concise notes to the account relating to any changes made to the Account, Groups, Projects etc.
• If the account is no longer considered abusive or in violation of our Terms of Use, remove any block or restrictions implemented on the account as a result of the Abuse.

The purpose of adding Admin Notes allow us to better assist the Support Team and Production if there are any questions around changes made to an account by the Security Team.

DMCA Requests

The Security Team plays a big role in defining the procedures and reviewing Digital Millennium Copyright Act (DMCA) requests. All DMCA requests need to be vetted by Legal first before any further steps are taken to proceed with the take down of reported content. Reported content that has been successfully vetted by Legal must be referred to the Abuse Team before any action is taken.

For DMCA requests the Abuse Team will follow the below process

• Confirm that the content is still live.
• Temporarlily remove the content to allow the parties to commence litigation.
• Return the content to the original state if no litigation was commenced.

Abuse works in conjunction with Legal referencing the DMCA Removal Workflow

Vulnerability Reports and HackerOne

GitLab receives vulnerability reports by various pathways, including:

• HackerOne bug bounty program
• security@gitlab.com
• Reports or questions come in from customers through Zendesk.
• Issues opened on the public issue trackers. The security team can not review all new issues and relies on everyone in the company to identify and label issues as ~security and @-mention @gitlab-com/gl-security/appsec on issues.

For any reported vulnerability:

• Open a confidential issue in the appropriate issue tracker as soon as a report is verified. If the vulnerability was reported via a public issue, make the issue confidential. If triage is delayed due to team availability, the delay should be communicated.
• An initial determination should be made as to severity and impact. Never dismiss a security report outright. Instead, follow up with the reporter, asking clarifying questions.
• For next steps, see the process as it is detailed below for HackerOne reports, and adhere to the guidelines there for vulnerabilities reported in other ways as well in terms of frequency of communication and so forth.
• Remember to prepare patches, blog posts, email templates, etc. on dev or in other non-public ways even if there is a reason to believe that the vulnerability is already out in the public domain (e.g. the original report was made in a public issue that was later made confidential).

HackerOne Process

GitLab utilizes HackerOne for its bug bounty program. Security researchers can report vulnerabilities in GitLab applications or the GitLab infrastructure via the HackerOne website. Team members authorized to respond to HackerOne reports use procedures outlined here.

The #hackerone-feed Slack channel receives notifications of report status changes and comments via HackerOne's Slack integration.

Triage Rotation

Application Security team members may assign themselves as the directly responsible individual (DRI) for incoming HackerOne reports for a given calendar week in the Triage Rotation Google Sheet in the Security Team Drive.

Duties:

• Point of contact for "New" HackerOne reports during that week.
• Responsible to escalating to other team members and management if the size of the queue spikes.

Team members should not assign themselves on weeks they are responsible for the scheduled security release.

Team members not assigned as the DRI for the week should continue to triage reports when possible, especially to close duplicates or handle related reports to those they have already triaged.

Team members remain responsible for their own assigned reports.

Working the Queue

• When beginning work on a report, the security team member should assign the report to themselves immediately.
• It is OK to take a report that you will not work on immediately, especially if it is a duplicate or related to another report you are familiar with, just be sure to get it reassigned if you won't be able to meet the estimated triage time.
• When starting a triage work cycle, team members should close duplicate and invalid reports first, then sort by "Oldest" reports. This helps reduce duplication of work and the import of duplicate reports. If a reporter has questions about the order of report responses, 06 - Duplicates/Invalid Reports Triaged First common response may be used.
• Conduct initial communication with the reporter and investigation, using the following guidelines as necessary:
  • Request clarification
  • Attempt to verify the report and triage the vulnerability.
• Potential, non-bounty outcomes:
  • Report is out-of-scope. If actionable, issues may still be created, but further process may not be followed see violating the rules.
  • Report is a ~feature as defined above and would not need to be made confidential or scheduled for remediation. An issue can be created, or requested that the reporter creates one if desired, but the report can be closed as "Informational".
  • Report is invalid. For example, we've gotten a number of reports for inline HTML in markdown that do not exceed the capabilities of markdown itself.
  • Report is a duplicate. Verify that the issue was not previously reported in HackerOne, or that an issue does not already exist in GitLab. If it is a duplicate:
    • Change the state of the report to "Duplicate".
    • If the issue was previously reported in H1, include the report id, as it can impact the reporter's reputation
    • Fill in the 01 - Duplicate common response. Include the link to the GitLab issue.
    • The team member may use their discretion if the reporter asks to be added as a contributor to the original H1 report; however, the default is to not add because the corresponding GitLab issue will be made public 30 days after the patch is released. If it is decided to add the duplicate reporter, ensure that the report does not have reporter sensitive information before allowing access. 05 - Duplicate Follow Up, No Adding Contributors/Future Public Release common response can be used when denying the request to add as a contributor.
• If the report is valid, in-scope, original, and requires action or further investigation by the responsible engineering team:
  • Verify and/or set the appropriate Severity in H1
  • Verify and/or set the appropriate Weakness in H1
  • Import the report into a GitLab issue using /h1import *report_id* [ce/ee] [Sx] [Px] in Slack
    • Assign the appropriate Severity and Priority.
    • Assign the appropriate Due Date
  • @ mention product manager of appropriate teams for scheduling and/or the engineering managers if additional engineering feedback is required to complete the triage, based on the current organization chart.
  • As applicable, notify relevant team members via the issue, chat, and email, depending on the chosen security level.
  • Change the state of the report to "Triaged" in HackerOne and include a link to the issue as the reference.
    • Fill in the comment using the 00 - Triaged Common response as a template.
    • In the comment, include link to the confidential issue
• Discussion and remediation of vulnerabilities can sometimes take longer than we would prefer. Even so, frequent communication with reporters is far better than leaving reporters in the dark, even if our progress is slow. Therefore:
  • Once the corresponding GitLab issue has been assigned a milestone, an automated process will add a comment to the H1 report with the planned fix date. If the issue has not been assigned to a milestone within 1 week of the product manager being @ mentioned on the issue, follow up with the product management team.
  • In the case where fixes are not as clear or discussion is still on-going as to whether a patch will be created at all, reporters should be notified of updates at least monthly.
  • Awards may be paid in advance of a confirmed fix by the following review process to insure 1) the report is not duplicated by other active reports and 2) consistency in award amounts:
    • Post a message on the appsec team Slack channel with a link to the report and a brief summary about the issue.
    • Include an initial proposed award amount and a brief reason.
    • Wait for at least two thumbs up from other members of the team.
    • The 04 - Bounty Award / Reviewed and Awarded Prior to Fix can be used as the template for the bounty award if approved.
  • In any case, no report should go "stale" where updates are not provided within the last month.

If a Report is Unclear

If a report is unclear, or the reviewer has any questions about the validity of the finding or how it can be exploited, now is the time to ask. Move the report to the "Needs More Info" state until the researcher has provided all the information necessary to determine the validity and impact of the finding. Use your best judgement to determine whether it makes sense to open a confidential issue anyway, noting in it that you are seeking more information from the reporter. When in doubt, err on the side of opening the issue.

One the report has been clarified, follow the "regular flow" described above.

If a Report Violates the Rules

If a report violates the rules of GitLab's bug bounty program use good judgement in deciding how to proceed. For instance, if a researcher has tested a vulnerability against GitLab production systems (a violation), but the vulnerability has not placed GitLab user data at risk, notify them that they have violated the terms of the bounty program but you are still taking the report seriously and will treat it normally. If the researcher has acted in a dangerous or malicious way, inform them that they have violated the terms of the bug bounty program and will not receive credit. Then continue with the "regular flow" as you normally would.

If the Report is Invalid

If the report is invalid (in your determination) or does not pose a security risk to GitLab or GitLab users it can be closed without opening an issue on GitLab.com. When this happens inform the researcher why it is not a vulnerability and close the issue as "Informational". HackerOne offers the option to close an issue as "Not Applicable" or "Spam". Both of these categories result in damage to the researcher's reputation and should only be used in obvious cases of abuse.

When a Patch is Ready

When a patch has been developed, tested, approved, merged into the security branch, and a new security release is being prepared it is time to inform the researcher via HackerOne. Post a comment on the HackerOne issue to all parties informing them that a patch is ready and will be included with the next security release. Provide release dates, if available, but try not to promise a release on a specific date if you are unsure.

This is also a good time to ask if they would like public credit in our release blog post and on our vulnerability acknowledgements page for the finding. We will link their name or alias to their HackerOne profile, Twitter handle, Facebook profile, company website, or URL of their choosing. Also ask if they would like the HackerOne report to be made public upon release. It is always preferable to publicly disclose reports unless the researcher has an objection.

CVE IDs

We use CVE IDs to uniquely identify and publicly define vulnerabilities in our products. Since we publicly disclose all security vulnerabilities 30 days after a patch is released, CVE IDs must be obtained for each vulnerability to be fixed. The earlier obtained the better, and it should be requested either during or immediately after a fix is prepared.

We currently request CVEs either through the HackerOne team or directly through MITRE's webform. Keep in mind that some of our security releases contain security related enhancements which may not have an associated CWE or vulnerability. These particular issues are not required to obtain a CVE since there's no associated vulnerability.

On Release Day

On the day of the security release several things happen in order:

• The new GitLab packages are published.
• All security patches are pushed to the public repository.
• The public is notified via the GitLab blog release post, security alerts email, and Twitter.
• The vulnerability acknowledgements page is updated with appropriate credits to the reporting researchers.

Once all of these things have happened notify the HackerOne researcher that the vulnerability and patch are now public. The GitLab issue should be closed and the HackerOne report should be closed as "Resolved". Public disclosure should be requested if they have not objected to doing so. Any sensitive information contained in the HackerOne report should be sanitized before disclosure.

Swag for Reports

GitLab awards swag codes for free GitLab swag to any reports that result in a security patch. Limit: 1 per reporter. When a report is closed, ask the reporter if they would like a swag code for free GitLab clothing or accessories. Swag codes are available by request from the marketing team.

Handling Disruptive Researcher Activity

Even though many of our 3rd-party dependencies, hosted services, and the static about.gitlab.com site are listed explicitly as out of scope, they are sometimes targeted by researchers. This results in disruption to normal GitLab operations. In these cases, if a valid email can be associated with the activity, a warning such as the following should be sent to the researcher using an official channel of communication such as ZenDesk.

Dear Security Researcher,

The system that you are accessing is currently out-of-scope for our bounty
program or has resulted in activity that is disruptive to normal GitLab
operations. Reports resulting from this activity may be disqualified from
receiving a paid bounty. Continued access to this system causing disuption to
GitLab operations, as described in policy under "Rules of Engagement,
Testing, and Proof-of-concepts", may result in additional restrictions on
participation in our program:

  Activity that is disruptive to GitLab operations will result in account bans and disqualification of the report.

Further details and some examples are available in the full policy available at:

https://hackerone.com/gitlab

Please contact us at security@gitlab.com with any questions.

Best Regards,

Security Department | GitLab
security@gitlab.com
https://about.gitlab.com/handbook/engineering/security/

External Code Contributions

We have a process in place to conduct security reviews for externally contributed code, especially if the code functionality includes any of the following:

• Processing credentials/tokens
• Storing credentials/tokens
• Logic for privilege escalation
• Authorization logic
• User/account access controls
• Authentication mechanisms

The Security Team works with our Community Outreach Team to ensure that security reviews are conducted where relevant. For more information about contributing, please reference the Contribute to GitLab page.

Security Questionnaires for Customers

Some customers, to keep up with regulations that impact their business, need to understand the security implications of installing any software - including software like GitLab.

Process

The current process for responding to customer requests is:

1. Refer a customer to our public statements on security here
2. If a customer still has questions that need to be discussed, you can engage a Solutions Architect in that discussion.
3. If the customer still needs a specific questionnaire filled out, create a confidential issue on the appropriate SA Triage board using the Vendor Security Assessment template with the label Security Audit and for the completion of that document
4. The SA team will take the first pass at the questionnaire using /security/ and this folder as a reference.
5. Once the SA team has completed what they can, the questionnaire will go to the security team for additional answers.
6. The security team requests ten (10) business days to complete their review. In many cases we can turn these around more quickly so every effort will be made to meet requested deadline.
7. Once the questionnaire is complete, it will need to be approved by the Director of Security for release to the customer.
8. File the completed questionnaire in the example folder for future reference.

Vulnerability Scanning

GitLab maintains a custom vulnerability scanner that is used to regularly scan all GitLab assets for common vulnerabilities as well as previously patched GitLab vulnerabilities and to ensure that no GitLab security-sensitive services are accidentally exposed.

Details on this scanner and how it is configured are available to all team members in a Google Doc entitled "Vulnerability Scanner Config".

Package Signing

The packages we ship are signed with GPG keys, as described in the omnibus documentation. The process around how to make and store the key pair in a secure manner is described in the runbooks. Those runbooks also point out that the management of the keys is handled by the Security team and not the Build team. For more details that are specific to key locations and access at GitLab, find the internal google doc titled "Package Signing Keys at GitLab" on Google Drive.