Secure Sub-Department

1. You are here:
2. Engineering
3. Development Department
4. Secure Sub-Department

Maintained by:

On this page

• Vision
• Mission
  • Be a good global citizen
• Team Members
  • Composition Analysis
  • Static Analysis
  • Dynamic Analysis
  • Fuzz Testing
  • Frontend Team
  • Vulnerability Research
• Stable Counterparts
• Secure Team
  • Security Products
  • Domains of Expertise
    • SAST
    • DAST
    • Dependency Scanning
    • Fuzz Testing
    • License Compliance
    • Vulnerability Research
  • Label Usage
  • Delineate Secure and Threat Management
  • Skills
  • Release process
  • QA process
  • Error Monitoring
  • Brainstorming sessions
  • Secure Architectural Council (Slack #s_secure-architectural-council)
    • Considerations when determining the approach
    • Capturing Resolved/Discovered Standards
    • Scope
    • Acceptance Criteria
    • Architectural Support Paste
  • Product Documentation
  • Async Daily Standups
    • Standup messages format
  • Recorded meetings
  • Calendar
    • Technical onboarding
    • Workflow and Refinement
• Coding standards and style guidelines
  • Cross group collaboration
  • Development of new analyzers
    • Checklist
      • Underlying tool
      • Dockerfile
      • Minimal vulnerability data
• Technical Documentation
• Retrospectives
• Common Links

The secure engineering sub-department is responsible for most of the secure stage of the product. (One portion of the secure stage (threat insights) is a responsibility of the Threat Management engineering sub-department).

Vision

To provide content and tools to support the best possible assessment at the earliest possible moment.

For more details about the vision for this area of the product, see the Secure stage page.

Mission

Be a good global citizen

We will give back to the Open Source community We will provide Responsible Disclosure We will share our knowledge and findings at public speaking events

The Secure team works on GitLab's Secure stage.

Team Members

The following people are permanent members of the Secure Sub-Department:

Composition Analysis

Static Analysis

Dynamic Analysis

Fuzz Testing

Frontend Team

Vulnerability Research

Stable Counterparts

The following members of other functional teams are our stable counterparts:

Secure Team

The Secure Team (previously known as the Security Products Team) is responsible for the security checks features in the GitLab platform, and maps to the secure transversal stage. You can learn more about our approach on the Secure Vision page.

The features provided by the Secure Team are mostly present at the pipeline level, and mostly available as Docker images. This particularity shapes our processes and QA, which differs a bit from the other backend teams.

Security Products

We still refer to "Security Products" as the tools developed by the Secure Team. Hence the home of our projects in GitLab: https://gitlab.com/gitlab-org/security-products/

Domains of Expertise

SAST

SAST (Static Application Security Testing) refers to static code analysis. GitLab leverages the power of various opensource tools to provide a wide range of checks for many languages and support. These tools are wrapped inside docker images which ensure we get a standard output from there. An orchestrator, developed by GitLab, is in charge of running these images, and gathering all the data needed to generate the final report.

DAST

DAST (Dynamic Application Security Testing) is used to hit a live application. Because some vulnerabilities can only be detected once all the code is actually running, this method complements the static code analysis. DAST is relying on OWASP Zed Attack Proxy Project, modified by GitLab to enable authentication.

Dependency Scanning

Dependency Scanning is used to detect vulnerabilities introduced by external dependencies in the application. Because a large portion of the code shipped to production is actually coming from third-party libraries, it's important to monitor them as well. Dependency Scanning is relying mostly on the Gemnasium engine.

Fuzz Testing

Coverage-guided fuzzing and API fuzzing are used to automatically input data into applications or web apis that has the potential to cause crashes or bugs. Coverage-guided fuzzing relies on open-sourced language-specific fuzzers. API Fuzzing is based on a proprietary GitLab engine.

License Compliance

License Compliance helps with the licenses introduced by third-party libraries in the application. Licence management relies on the LicenseFinder gem.

Vulnerability Research

The Vulnerability Research team's purpose is to perform research and develop proofs of concepts that increase the capabilities and effectiveness of the Secure stage.

Label Usage

If you are submitting an issue about a Secure Stage feature, use ~devops::secure and one of the following group labels to get the issue in front of the most appropriate team members.

Additional labels should be added according to the Workflow Labels Documentation.

Delineate Secure and Threat Management

It is important to delineate who the EM and PM DRIs are for every functionality, especially where this may not be obvious. It is documented on a dedicated delineation page.

Skills

Because we have a wide range of domains to cover, it requires a lot of different expertises and skills:

Our team also must have a good sense of security, with at least basic skills in application security.

We provide tools for many different languages (ex: sast, dependency scanning, license compliance). It means our team is able to understand the basics of each of these languages, including their package managers. We maintain tests projects to ensure our features are working release after release for each of them.

Release process

See Secure Release Process.

QA process

See QA Process for more info.

Error Monitoring

500 errors on gitlab.com are reported to Sentry. Below are some quick links to pull up Sentry errors related to Secure.

• StoreSecurityReports Worker - https://sentry.gitlab.net/gitlab/gitlabcom/?query=is%3Aunresolved+StoreSecurityReportsWorker&statsPeriod=14d
• SyncSecurityReportsToReportApprovalRules Worker - https://sentry.gitlab.net/gitlab/gitlabcom/?query=is%3Aunresolved+SyncSecurityReportsToReportApprovalRulesWorker&statsPeriod=14d
• Vulnerabilities - https://sentry.gitlab.net/gitlab/gitlabcom/?query=is%3Aunresolved+vulnerabilities&statsPeriod=14d
• On-Demand DAST - https://sentry.gitlab.net/gitlab/gitlabcom/?query=is%3Aunresolved+Dast&statsPeriod=14d

Brainstorming sessions

Our team occasionally schedules synchronous brainstorming sessions as a method of deep-diving on a specific topic. This approach can be useful in breaking down complexity and deriving actionable steps for problems that lack definition.

These are purposefully freeform to allow for creative problem solving. When possible, time should be reserved for a list of actions to be taken from the open discussion.

Brainstorming Sessions Doc (Internal): https://docs.google.com/document/d/179JL5RzbgSIz2XZewbYn79cuX7_vUtte_TcoLwUUC5o/edit#

Examples of previous brainstorming topics:

• Reduce false positives in Security reports
• How to manage occurrence uniqueness identification in the common report format? (CompareKey)
• One file with syntax errors should not stop SAST and similar kind of jobs from running

Secure Architectural Council (Slack #s_secure-architectural-council)

In order to help the Sec section come to resolution on defining approaches to section-wide issues, we have formed an Architectural Council. This group comprises tech leads (and SMEs to provide context) in order to keep the team small and nimble. The group is a non-authoritative, supportive group whose members provide representative insights from their respective teams.

• Tech leads will serve as the representative for their team
• An issue should be created to capture the discussion that covers:
  • What is the issue at hand and what is the preferred action
  • What are the potential solutions and their associated pros/cons
  • What approach was decided and why
  • 3-business day SLO
    • We need to know when to start the clock on these as some issues can exist for months before being brought to the council. In this case, we should define within the issue when the clock does start.
• There will be a DRI assigned to the issue that is being discussed
• The DRI will be decided by either who has to address the issue first or where the code change is occurring the most (in that order)
• The issue will be discussed and the DRI will be the ultimate decision-maker for the approach taken
• Issues will be tackled on a First-In-and-First-Out (FIFO) order. Attempts to minimize SLA overlap should be made to prevent scheduling conflicts between member’s time

Considerations when determining the approach

• Simplicity and elegance
• Portability and/or modularity
• Supportability
• Maintainability
• Scalability
• Extensibility
• Reliability
• Security
• Cost (will it impact .com for large customers)
• Performance (speed and accuracy)
• Consistency (Fit with existing code base)

Capturing Resolved/Discovered Standards

Common scenarios/architectural concepts will likely be resolved/discovered. These should be captured generically in an architectural guidelines page of the handbook and, if possible, codified into our processes/templates.

Scope

The table below captures characteristics (requirements?) of work that is in-scope, opt-in, or out-of-scope. All requirements must be met for each category.

Acceptance Criteria

GitLab’s Stance for Architectural issues: https://about.gitlab.com/handbook/engineering/architecture/

Architectural Support Paste

Product Documentation

As the product evolves, it is important to maintain accurate and up to date documentation for our users. If it is not documented, customers may not know a feature exists.

To update the documentation, the following process should be followed:

1. When an issue has been identified as needing documentation, add the ~Documentation label, outline in the description of the issue what documentation is needed, and assign a Backend Engineer and Technical Writer(TW) to the issue (find the appropriate TW by searching the product categories).
2. If the task is documentation only, apply a ~Px label.
3. For documentation around features or bugs, a backend engineer should write the documentation and work with the technical writer for editing. If the documentation only needs styling cleanup, clarification, or reorganization, this work should be lead by the Technical Writer with support from a BE as necessary. The availability of a technical writer should in no way hold up work on the documentation. Further information on the documentation process.

Async Daily Standups

Since we are a remote company, having daily standup meetings would not make any sense, since we're not all in the same timezone. That's why we have async daily standups, where everyone can give some insights into what they did yesterday, what they plan to do today, etc. For that, we rely on the geekbot slack plugin to automate the process.

Standup messages format

• Use the "description in backquote + [link to issue](#)" format when mentioning issues in your standup report.
• Prepend CI status icons to the answer lines for What did you do since yesterday? to denote the current state:
  • for successfully accomplished tasks (:ci_passing: emoji)
  • for tasks that were due on some period of time but were not accomplished (:ci_failing: emoji)
  • for tasks currently in progress (:ci_running: emoji)
  • for paused or postponed tasks (:ci_pending: emoji)
  • any other :ci_... icon you find applicable

Example:

What did you do since yesterday?

• Accomplished the Spotbugs java analyzer compareKey is not unique https://gitlab.com/gitlab-org/gitlab-ee/issues/10860
• Still working on Allow guests to create an issue from a vulnerability https://gitlab.com/gitlab-org/gitlab-ee/issues/7813
• Catch-up on all emails and threads after the vacation

Slack Channels:

As our teams focus on different areas, we have Geekbot configured to broadcast to separate channels in addition to our common one at #s_secure-standup.

1. Composition Analysis: #g_secure-composition-analysis-standup
2. Static Analysis: #g_secure-static-analysis-standup
3. Dynamic Analysis: #g_secure-dynamic-analysis-standup
4. Frontend: #s_secure-frontend-standup

Recorded meetings

Our important meetings are recorded and published on YouTube, in the GitLab Secure Stage playlist. They give a good overview of the decision process, which is often a discussion with all the stakeholders. As we are a remote company, these video meetings help to synchronize and take decisions faster than commenting on issues. We prefer asynchronous work, but for large features and when the timing is tight, we can detail a lot of specifications. This will make the asynchronous work easier, since we have evaluated all edge cases.

Calendar

We welcome team members to join meetings that are on our shared calendar. The Secure Calendar is available to all logged in GitLab team members.

Technical onboarding

New hires should go through these steps and read the corresponding documentation when onboarding in the Secure Team. Every new hire will have an assigned onboarding issue that will guide them through the whole process.

Workflow and Refinement

Secure largely follows our Product Development Flow

See Issue Refinement to learn how we evaluate complexity, level of effort, our implementation plan and assign issue weights.

Coding standards and style guidelines

The Secure Team follows the coding standards and style guidelines outlined in the company-wide Contributor and Development Docs, however, please consult the following guidelines which are specific to the Secure Team:

• Secure Team Go standards and style guidelines

Cross group collaboration

Some components of the architecture that support Secure features are shared between multiple groups like the common Go library, the Security Report Schemas, the rails parsers, etc.

Modifying these shared pieces might impact other groups so we should rely as much as possible on approval rules to ensure such changes are reviewed by the relevant teams before being merged.

Impactless two-way door changes could skip the approval process, please use sound judgement and common sense in such situations.

The author of changes should announce broadly the changes made on these components to raise awareness (weekly meeting agenda, slack channel).

Development of new analyzers

We occasionally need to build out new analyzer projects to support new frameworks and tools. In doing so we should follow our engineering Open Source guidelines, including licensing and code standards.

In addition, to write a custom analyzer that will integrate into the GitLab application a minimal featureset is required:

Checklist

Underlying tool

• Has permissive software license
• Headless execution (CLI tool)
• Bundle-able dependencies to be packaged as a Docker image, to be executed using GitLab Runner's Linux or Windows Docker executor
• Compatible projects can be detected based on filenames or extensions
• Offline execution (no Internet access) or can be configured to use custom proxies and/or CA certificates

Dockerfile

The Dockerfile should use an unprivileged user with the name gitlab. The reason this is necessary is to provide compatibility with Red Hat OpenShift instances, which don't allow containers to run as an admin (root) user. There are certain limitations to keep in mind when running a container as an unprivileged user, such as the fact that any files that need to be written on the Docker filesystem will require the appropriate permissions for the gitlab user. Please see the following merge request for more details: Use gitlab user instead of root in Docker image.

Minimal vulnerability data

Please see our security-report-schemas for a full list of required fields.

The security-report-schema repository contains JSON schemas that list the required fields for each report type:

• Container Scanning
• DAST
• Dependency Scanning
• SAST
• Secret Detection

Technical Documentation

As our product evolves, the engineering teams are researching ways to achieve new functionality and improve our architecture.

The outcome of this research can be found in our Technical Documentation section.

Retrospectives

The Secure sub-department conducts retrospectives at the group level that follow our engineering workflow. Each group's DRI is responsible to prepare and schedule the retrospective sync sessions and the async retrospective issues can be found in the corresponding project.

NB: we use to have a sub-department wide retrospective whose issues are still accessible in the deprecated project.

Common Links

• Secure team board
• #s_secure in Slack
• Secure and Protect Glossary of Terms