Secure Sub-Department

1. You are here:
2. Handbook
3. Engineering
4. Development Department
5. Secure Sub-Department

Maintained by:

Todd Stadelhofer

@tstadelhofer

On this page

• Vision
• Mission
  • Do no harm
  • Be a good global citizen
• Team Members
  • Composition Analysis
  • Static Analysis
  • Dynamic Analysis
  • Frontend Team
  • Vulnerability Research
• Stable Counterparts
• Secure Team
  • Security Products
  • Domains of Expertise
    • SAST
    • DAST
    • Dependency Scanning
    • Container Scanning
    • License Compliance
    • Vulnerability Research
  • Label Usage
  • Delineate Secure and Threat Management
  • Skills
  • Release process
  • QA process
  • Error Monitoring
  • Brainstorming sessions
  • Product Documentation
  • Async Daily Standups
    • Standup messages format
  • Recorded meetings
  • Calendar
    • Technical onboarding
    • Workflow and Refinement
• Coding standards and style guidelines
  • Cross group collaboration
  • Development of new analyzers
    • Checklist
      • Underlying tool
      • Minimal vulnerability data
• Technical Documentation
• Retrospectives
  • Rotation
• Common Links

The secure engineering sub-department is responsible for most of the secure stage of the product. (One portion of the secure stage (threat insights) is a responsibility of the Threat Management engineering sub-department).

Vision

To provide content and tools to support the best possible assessment at the earliest possible moment.

For more details about the vision for this area of the product, see the Secure stage page.

Mission

Do no harm

We will never knowingly compromise the security posture of your application.

Be a good global citizen

We will give back to the Open Source community We will provide Responsible Disclosure We will share our knowledge and findings at public speaking events

The Secure team works on GitLab's Secure stage.

Team Members

The following people are permanent members of the Secure Sub-Department:

Composition Analysis

Static Analysis

Dynamic Analysis

Frontend Team

Vulnerability Research

Stable Counterparts

The following members of other functional teams are our stable counterparts:

Secure Team

The Secure Team (previously known as the Security Products Team) is responsible for the security checks features in the GitLab platform, and maps to the secure transversal stage. You can learn more about our approach on the Secure Vision page.

The features provided by the Secure Team are mostly present at the pipeline level, and mostly available as Docker images. This particularity shapes our processes and QA, which differs a bit from the other backend teams.

Security Products

We still refer to "Security Products" as the tools developed by the Secure Team. Hence the home of our projects in GitLab: https://gitlab.com/gitlab-org/security-products/

Domains of Expertise

SAST

SAST (Static Application Security Testing) refers to static code analysis. GitLab leverages the power of various opensource tools to provide a wide range of checks for many languages and support. These tools are wrapped inside docker images which ensure we get a standard output from there. An orchestrator, developed by GitLab, is in charge of running these images, and gathering all the data needed to generate the final report.

DAST

DAST (Dynamic Application Security Testing) is used to hit a live application. Because some vulnerabilities can only be detected once all the code is actually running, this method complements the static code analysis. DAST is relying on OWASP Zed Attack Proxy Project, modified by GitLab to enable authentication.

Dependency Scanning

Dependency Scanning is used to detect vulnerabilities introduced by external dependencies in the application. Because a large portion of the code shipped to production is actually coming from third-party libraries, it's important to monitor them as well. Dependency Scanning is relying mostly on the Gemnasium engine.

Container Scanning

Container Scanning is used when the application is shipped as a Docker image. It's very common to build the final image on top of an existing one, which must be checked like every other portion of the application. For that, Container Scanning is relying on klar and clair server.

License Compliance

License Compliance helps with the licenses introduced by third-party libraries in the application. Licence management relies on the LicenseFinder gem.

Vulnerability Research

The Vulnerability Research team's purpose is to perform research and develop proofs of concepts that increase the capabilities and effectiveness of the Secure stage.

Label Usage

If you are submitting an issue about a Secure Stage feature, use ~devops::secure and one of the following group labels to get the issue in front of the most appropriate team members.

Additional labels should be added according to the Workflow Labels Documentation.

Delineate Secure and Threat Management

It is important to delineate who the EM and PM DRIs are for every functionality, especially where this may not be obvious. It is documented on a dedicated delineation page.

Skills

Because we have a wide range of domains to cover, it requires a lot of different expertises and skills:

Our team also must have a good sense of security, with at least basic skills in application security.

We provide tools for many different languages (ex: sast, dependency scanning, license compliance). It means our team is able to understand the basics of each of these languages, including their package managers. We maintain tests projects to ensure our features are working release after release for each of them.

Release process

See Secure Release Process.

QA process

Our release scripts project also defines our QA process.

Error Monitoring

500 errors on gitlab.com are reported to Sentry. Below are some quick links to pull up Sentry errors related to Secure.

• StoreSecurityReports Worker - https://sentry.gitlab.net/gitlab/gitlabcom/?query=is%3Aunresolved+StoreSecurityReportsWorker&statsPeriod=14d
• SyncSecurityReportsToReportApprovalRules Worker - https://sentry.gitlab.net/gitlab/gitlabcom/?query=is%3Aunresolved+SyncSecurityReportsToReportApprovalRulesWorker&statsPeriod=14d
• Vulnerabilities - https://sentry.gitlab.net/gitlab/gitlabcom/?query=is%3Aunresolved+vulnerabilities&statsPeriod=14d

Brainstorming sessions

Our team occasionally schedules synchronous brainstorming sessions as a method of deep-diving on a specific topic. This approach can be useful in breaking down complexity and deriving actionable steps for problems that lack definition.

We tend to schedule these on a weekly cadence with a rotating timeslot to accommodate team members in different timezones. If a topic is not agreed upon prior to a given week, the slot is canceled to avoid unnecessary meetings. These slots can be viewed on the Secure Stage Team Calendar.

These are purposefully freeform to allow for creative problem solving. When possible, time should be reserved for a list of actions to be taken from the open discussion.

Brainstorming Sessions Doc (Internal): https://docs.google.com/document/d/179JL5RzbgSIz2XZewbYn79cuX7_vUtte_TcoLwUUC5o/edit#

Examples of previous brainstorming topics:

• Reduce false positives in Security reports
• How to manage occurrence uniqueness identification in the common report format? (CompareKey)
• One file with syntax errors should not stop SAST and similar kind of jobs from running

Product Documentation

As the product evolves, it is important to maintain accurate and up to date documentation for our users. If it is not documented, customers may not know a feature exists.

To update the documentation, the following process should be followed:

1. When an issue has been identified as needing documentation, add the ~Documentation label, outline in the description of the issue what documentation is needed, and assign a Backend Engineer and Technical Writer(TW) to the issue (find the appropriate TW by searching the product categories).
2. If the task is documentation only, apply a ~Px label.
3. For documentation around features or bugs, a backend engineer should write the documentation and work with the technical writer for editing. If the documentation only needs styling cleanup, clarification, or reorganization, this work should be lead by the Technical Writer with support from a BE as necessary. The availability of a technical writer should in no way hold up work on the documentation. Further information on the documentation process.

Async Daily Standups

Since we are a remote company, having daily standup meetings would not make any sense, since we're not all in the same timezone. That's why we have async daily standups, where everyone can give some insights into what they did yesterday, what they plan to do today, etc. For that, we rely on the geekbot slack plugin to automate the process.

Standup messages format

• Use the "description in backquote + [link to issue](#)" format when mentioning issues in your standup report.
• Prepend CI status icons to the answer lines for What did you do since yesterday? to denote the current state:
  • for successfully accomplished tasks (:ci_passing: emoji)
  • for tasks that were due on some period of time but were not accomplished (:ci_failing: emoji)
  • for tasks currently in progress (:ci_running: emoji)
  • for paused or postponed tasks (:ci_pending: emoji)
  • any other :ci_... icon you find applicable

Example:

What did you do since yesterday?

• Accomplished the Spotbugs java analyzer compareKey is not unique https://gitlab.com/gitlab-org/gitlab-ee/issues/10860
• Still working on Allow guests to create an issue from a vulnerability https://gitlab.com/gitlab-org/gitlab-ee/issues/7813
• Catch-up on all emails and threads after the vacation

Slack Channels:

As our teams focus on different areas, we have Geekbot configured to broadcast to separate channels in addition to our common one at #s_secure-standup.

1. Composition Analysis: #g_secure-composition-analysis-standup
2. Static Analysis: #g_secure-static-analysis-standup
3. Dynamic Analysis: #g_secure-dynamic-analysis-standup
4. Frontend: #s_secure-frontend-standup

Recorded meetings

Our important meetings are recorded and published on YouTube, in the GitLab Secure Stage playlist. They give a good overview of the decision process, which is often a discussion with all the stakeholders. As we are a remote company, these video meetings help to synchronize and take decisions faster than commenting on issues. We prefer asynchronous work, but for large features and when the timing is tight, we can detail a lot of specifications. This will make the asynchronous work easier, since we have evaluated all edge cases.

Calendar

We welcome team members to join meetings that are on our shared calendar. The Secure Calendar is available to all logged in GitLab team members.

Technical onboarding

New hires should go through these steps and read the corresponding documentation when onboarding in the Secure Team. Every new hire will have an assigned onboarding issue that will guide them through the whole process.

Workflow and Refinement

See Secure Workflow and Issue Refinement.

Coding standards and style guidelines

The Secure Team follows the coding standards and style guidelines outlined in the company-wide Contributor and Development Docs, however, please consult the following guidelines which are specific to the Secure Team:

• Secure Team Go standards and style guidelines

Cross group collaboration

Some components of the architecture that support Secure features are shared between multiple groups like the common Go library, the Security Report Schemas, the rails parsers, etc.

Modifying these shared pieces might impact other groups so we should rely as much as possible on approval rules to ensure such changes are reviewed by the relevant teams before being merged.

Impactless two-way door changes could skip the approval process, please use sound judgement and common sense in such situations.

The author of changes should announce broadly the changes made on these components to raise awareness (weekly meeting agenda, slack channel).

Development of new analyzers

We occasionally need to build out new analyzer projects to support new frameworks and tools. In doing so we should follow our engineering Open Source guidelines, including licensing and code standards.

In addition, to write a custom analyzer that will integrate into the GitLab application a minimal featureset is required:

Checklist

Underlying tool

• Has permissive software license
• Headless execution (CLI tool)
• Bundle-able dependencies to be packaged as a Docker image, to be executed using GitLab Runner's Linux or Windows Docker executor
• Compatible projects can be detected based on filenames or extensions
• Offline execution (no Internet access) or can be configured to use custom proxies and/or CA certificates

Minimal vulnerability data

Please see our security-report-schemas for a full list of required fields.

The security-report-schema repository contains JSON schemas that list the required fields for each report type:

• Container Scanning
• DAST
• Dependency Scanning
• SAST
• Secret Detection

Technical Documentation

As our product evolves, the engineering teams are researching ways to achieve new functionality and improve our architecture.

The outcome of this research can be found in our Technical Documentation section.

Retrospectives

The Secure Team conducts a department wide retrospective on the 24th of each month. You can see previous topics within our past Secure team retrospectives issues.

Each month an engineering manager from one of our teams acts as the DRI for the retrospective. This individual schedules sessions and captures our focus outcomes to acheive in the next milestone.

Rotation

• Neil McCorrison - 13.1
• Seth Berger - 13.2
• Olivier Gonzalez - 13.3
• Thomas Woodham - 13.4

Retro handoff process

To provide a soft handoff betwen milestones we rely on the current retro DRI to take a few steps. On the 1st of the month:

1. Confirm who is leading the next retrospective.
2. Reschedule/create our EMEA and APAC retro sync sessions if necessary.
3. Ensure previous retro focus outcomes have associated issues and DRIs.

Common Links

• Secure team board
• #s_secure in Slack
• Secure and Defend Glossary of Terms