Secure Sub-Department
The Secure engineering sub-department is responsible for the Secure Stage of the product.
Vision
To provide content and tools to support the best possible assessment at the earliest possible moment.
Following our single application paradigm, we integrate and build scanning tools to supply security and compliance assessment data to the main GitLab application where we develop our vulnerability management system and other features. While it might be technically feasible, we do not aim at building standalone products that provide this data independently from the GitLab application.
For more details about the vision for this area of the product, see the Secure stage page.
Mission
To support the success of GitLab by developing highly usable, hiqh quality tools for customers to build more secure software.
The Secure team works on GitLab’s Secure stage.
Sub-department development people leaders
Name | Role |
---|---|
Thomas Woodham | Senior Engineering Manager, Secure |
Alex Groleau | Engineering Manager, Secure:Dynamic Analysis |
Amar Patel | Engineering Manager, Secure:Secret Detection |
Tal Kopel | Manager, Software Engineering, Secure:Composition Analysis |
Thiago Figueiró | Manager, Software Engineering, Secure:Composition Analysis |
To contact Secure stage development people leaders leaders use the following aliases:
- GitLab:
@gitlab-org/secure/managers
- Slack:
@s_secure_managers
Team Members
The following people are permanent members of the Secure Sub-Department:
Composition Analysis
Team Page: Composition Analysis
Name | Role |
---|---|
Aditya Tiwari | Senior Backend Engineer, Secure:Composition Analysis |
Fabien Catteau | Staff Backend Engineer, Secure:Composition Analysis |
Igor Frenkel | Senior Backend Engineer, Secure:Composition Analysis |
Miki Amos | Senior Backend Engineer, Secure:Composition Analysis |
Nick Ilieskou | Senior Backend Engineer, Secure:Composition Analysis |
Olivier Gonzalez | Staff Backend Engineer, Secure:Composition Analysis |
Orin Naaman | Backend Engineer, Secure:Composition Analysis |
Oscar Tovar | Senior Backend Engineer, Secure:Composition Analysis |
Philip Cunningham | Senior Backend Engineer, Secure:Composition Analysis |
Shao Ming Tan | Backend Engineer, Secure:Composition Analysis |
Tal Kopel | Manager, Software Engineering, Secure:Composition Analysis |
Thiago Figueiró | Manager, Software Engineering, Secure:Composition Analysis |
Yasha Rise | Senior Backend Engineer, Secure:Composition Analysis |
Zamir Martins Filho | Senior Fullstack Engineer, Secure:Composition Analysis |
Name | Role |
---|---|
Fernando Cardenas | Senior Frontend Engineer, Secure:Composition Analysis |
Static Analysis
Team Page: Static Analysis
Name | Role |
---|
Dynamic Analysis
Team Page: Dynamic Analysis
Name | Role |
---|---|
Thomas Woodham | Senior Engineering Manager, Secure |
Alex Groleau | Engineering Manager, Secure:Dynamic Analysis |
Vulnerability Research
Team Page: Vulnerability Research
Name | Role |
---|
Stable Counterparts
The following members of other functional teams are our stable counterparts:
Name | Role |
---|---|
Ahmed Hemdan | Senior Backend Engineer, Secure:Secret Detection |
Connor Gilbert | Principal Product Manager, Secure:Static Analysis |
Dheeraj Joshi | Senior Frontend Engineer, Secure:Secret Detection |
Nikhil George | Senior Security Engineer, Application Security, Secure (Static Analysis, Dynamic Analysis, Composition Analysis, Vulnerability Research), Growth (Acquisition, Activation). |
Ross Fuhrman | Senior Backend Engineer, Secure:Secret Detection |
Russell Dickenson | Senior Technical Writer, Secure |
Sarah Waldner | Group Manager, Product Management, Secure |
Serena Fang | Backend Engineer, Secure:Secret Detection |
Sara Meadzinger | Senior Product Manager, Secure:Dynamic Analysis |
Thomas Woodham | Senior Engineering Manager, Secure |
Vishwa Bhat | Backend Engineer, Secure:Secret Detection |
Will Meek | Senior Software Engineer in Test, Secure |
Secure Team
The Secure Team (previously known as the Security Products Team) is responsible for the security checks features in the GitLab platform, and maps to the secure transversal stage. You can learn more about our approach on the Secure Vision page.
The features provided by the Secure Team are mostly present at the pipeline level, and mostly available as Docker images. This particularity shapes our processes and QA, which differs a bit from the other backend teams.
Security Products
We still refer to “Security Products” as the tools developed by the Secure Team. Hence the home of our projects in GitLab: https://gitlab.com/gitlab-org/security-products/.
We strive to maintain a consistent User Experience across our Security Products but we do not enforce consistency at the implementation level. Each group faces its own challenges and is in the best position to make the technical choices it deems are the most suitable to achieve its goals. While UX inconsistencies are considered as bugs, we rely on individual teams to make smart decisions about when consistency is important and when divergence makes more sense — either because the divergence itself creates a better experience or because of velocity considerations.
Domains of Expertise
SAST
SAST (Static Application Security Testing) refers to static code analysis. GitLab leverages the power of various opensource tools to provide a wide range of checks for many languages and support. These tools are wrapped inside docker images which ensure we get a standard output from there. An orchestrator, developed by GitLab, is in charge of running these images, and gathering all the data needed to generate the final report.
DAST
DAST (Dynamic Application Security Testing) is used to hit a live application. Because some vulnerabilities can only be detected once all the code is actually running, this method complements the static code analysis. DAST is relying on OWASP Zed Attack Proxy Project, modified by GitLab to enable authentication.
Dependency Scanning
Dependency Scanning is used to detect vulnerabilities introduced by external dependencies in the application. Because a large portion of the code shipped to production is actually coming from third-party libraries, it’s important to monitor them as well. Dependency Scanning is relying mostly on the Gemnasium engine.
Fuzz Testing
Coverage-guided fuzzing and API fuzzing are used to automatically input data into applications or web apis that has the potential to cause crashes or bugs. Coverage-guided fuzzing relies on open-sourced language-specific fuzzers. API Fuzzing is based on a proprietary GitLab engine.
License Compliance
License Compliance helps with the licenses introduced by third-party libraries in the application. Licence management relies on the LicenseFinder gem.
Vulnerability Research
The Vulnerability Research team’s purpose is to perform research and develop proofs of concepts that increase the capabilities and effectiveness of the Secure stage.
Label Usage
If you are submitting an issue about a Secure Stage feature, use ~devops::secure
and one of the following group labels to get the issue in front of the most appropriate team members.
Label | Use |
---|---|
~devops::secure |
All issues related to the Secure Stage |
Additional labels should be added according to the Workflow Labels Documentation.
Skills
Because we have a wide range of domains to cover, it requires a lot of different expertises and skills:
Technology skills | Areas of interest |
---|---|
Ruby on Rails | Backend development |
Go | SAST, Dependency Scanning, DAST |
Python | DAST |
SQL (PostgreSQL) | Dependency Scanning / all |
Docker | Container Scanning / all |
C# | API Security |
Our team also must have a good sense of security, with at least basic skills in application security.
We provide tools for many different languages (ex: sast, dependency scanning, license compliance). It means our team is able to understand the basics of each of these languages, including their package managers. We maintain tests projects to ensure our features are working release after release for each of them.
Release process
See Versioning and release process.
QA process
See QA Process for more info.
Vulnerability Management process
Automation
We use the security-triage-automation tool in conjunction with scheduled pipelines in the release project to handle the following tasks:
- Create security issues for FedRAMP vulnerabilities (Container Scanning results only) still detected on the default branch , executed at least once, on the first day of the month to match with FedRAMP compliance report cadence. Note that we do not yet automatically create security issues for non-FedRAMP vulnerabilities. Please see the Non-FedRAMP vulnerabilities section for more details.
- Resolve all vulnerabilities (both FedRAMP and non-FedRAMP) no longer detected on the default branch and close their issues, executed every 2 days.
The Vulnmapper tool also provides some automation to vulnerability management like:
- Adding labels to security issues to further classify the fix availability (fix_available, fix_unavailable, will_not_be_fixed, etc.).
- Creating Deviation Request issues for FedRAMP related security issues that should have one.
Note: Our goal is to centralize automation for vulnerability management in the Vulnmapper tool in the nearest future and standardize our processes across the company. However, so far we’re following the existing process based on the security-triage-automation tool.
Automation failures
It’s possible that our security automation tooling may fail.
If this occurs, and the issue cannot be immediately resolved, open an issue to
track the error. Then, announce the failure in #s_secure
to raise awareness,
and follow the manual security triage process outlined below.
View manual process fallback when automation fails
Manually reviewing and resolving vulnerabilities
On a weekly basis: review the vulnerability report to resolve no longer detected ones and close related issues. Note: It is not necessary to investigate vulnerabilities that are no longer detected.
- Visit
Vulnerability Report Dashboards
to verify that there are vulnerabilities that can be resolved. - Execute the
security-triage-automation
tool to resolve vulnerabilities and close their issues. This tool must be executed separately for each project that have vulnerabilities to resolve. - Verify in
Vulnerability Report Dashboards
that vulnerabilities have been resolved.
Manually creating security issues for FedRAMP vulnerabilities
Last working day before the 1st of the month, create security issues
for FedRAMP vulnerabilities of the CONTAINER_SCANNING
type, and CRITICAL
, HIGH
,
MEDIUM
, LOW
, and UNKNOWN
severity levels by executing the security-triage-automation
tool to process vulnerabilities for a given project
(please make sure to adjust CLI options accordingly). This tool must be executed
separately for each project.
Manually creating deviation requests for FedRAMP vulnerabilities
In cases where automation fails, you must create the Deviation Requests manually before the issues reach SLA. To do so, use the following procedure.
-
Open a DR issue with the operational requirement template.
- Update the
Vulnerability Details
section with a link to the advisory (RedHat tracker usually), CVE ID, severity, and CVSS score. - Update the
Justification Section
with:The OS vendor has published an updated advisory for <CVE_ID>, indicating that package <PACKAGE_NAME> has not yet had a fix released for this vulnerability. Until a fix is available for the package, this vulnerability cannot practically be remediated.
- Update the
Attached Evidence
section with:As this operational requirement represents a dependency on a vendor-published package to address this vulnerability, no additional evidence has been supplied. Please refer to the linked vendor advisory in the above justification.
- Link it to the security issue:
/relate <issue_id>
- Update the
-
Update the security issue accordingly
/label ~"FedRAMP::Vulnerability" ~"FedRAMP::DR Status::Open" /milestone %Backlog
FedRAMP vulnerabilities
To ensure compliance, the management of FedRAMP vulnerabilities is handled by automation. Please check the manual process fallback for additional details.
Non-FedRAMP vulnerabilities
We do not yet have the same automation in place for non-FedRAMP vulnerabilities since it represents a too important volume to manage for our teams and some necessary improvements in the vulnmapper tool are required prior to enabling this. In the meantime, we favor a more specialized approach for these vulnerabilities and there is no standardized process across the groups.
Error Monitoring
500 errors on gitlab.com are reported to Sentry. Below are some quick links to pull up Sentry errors related to Secure.
- StoreSecurityReports Worker - https://sentry.gitlab.net/gitlab/gitlabcom/?query=is%3Aunresolved+StoreSecurityReportsWorker&statsPeriod=14d
- SyncSecurityReportsToReportApprovalRules Worker - https://sentry.gitlab.net/gitlab/gitlabcom/?query=is%3Aunresolved+SyncSecurityReportsToReportApprovalRulesWorker&statsPeriod=14d
- Vulnerabilities - https://sentry.gitlab.net/gitlab/gitlabcom/?query=is%3Aunresolved+vulnerabilities&statsPeriod=14d
- On-Demand DAST - https://sentry.gitlab.net/gitlab/gitlabcom/?query=is%3Aunresolved+Dast&statsPeriod=14d
Brainstorming sessions
Our team occasionally schedules synchronous brainstorming sessions as a method of deep-diving on a specific topic. This approach can be useful in breaking down complexity and deriving actionable steps for problems that lack definition.
These are purposefully freeform to allow for creative problem solving. When possible, time should be reserved for a list of actions to be taken from the open discussion.
Brainstorming Sessions Doc (Internal): https://docs.google.com/document/d/179JL5RzbgSIz2XZewbYn79cuX7_vUtte_TcoLwUUC5o/edit#
Examples of previous brainstorming topics:
- Reduce false positives in Security reports
- How to manage occurrence uniqueness identification in the common report format? (CompareKey)
- One file with syntax errors should not stop SAST and similar kind of jobs from running
Resources
- How to triage a QA test pipeline failure
- Beginner’s guide to writing end-to-end tests
- GitLab QA README
- GitLab QA Scenarios
- E2E info for GitLab developers
- Quality training video material
Product Documentation
As the product evolves, it is important to maintain accurate and up to date documentation for our users. If it is not documented, customers may not know a feature exists.
To update the documentation, the following process should be followed:
- When an issue has been identified as needing documentation, add the
~Documentation
label, outline in the description of the issue what documentation is needed, and assign a Backend Engineer and Technical Writer(TW) to the issue (find the appropriate TW by searching the product categories). - If the task is documentation only, apply a
~Px
label. - For documentation around features or bugs, a backend engineer should write the documentation and work with the technical writer for editing. If the documentation only needs styling cleanup, clarification, or reorganization, this work should be lead by the Technical Writer with support from a BE as necessary. The availability of a technical writer should in no way hold up work on the documentation. Further information on the documentation process.
Async Daily Standups
Since we are a remote company, having daily standup meetings would not make any sense, since we’re not all in the same timezone. That’s why we have async daily standups, where everyone can give some insights into what they did yesterday, what they plan to do today, etc. For that, we rely on the geekbot slack plugin to automate the process.
Standup messages format
- Use the “
description in backquote
+[link to issue](#)
” format when mentioning issues in your standup report. - Prepend CI status icons to the answer lines for
What did you do since yesterday?
to denote the current state:- for successfully accomplished tasks (
:ci_passing:
emoji) - for tasks that were due on some period of time but were not accomplished (
:ci_failing:
emoji) - for tasks currently in progress (
:ci_running:
emoji) - for paused or postponed tasks (
:ci_pending:
emoji) - any other
:ci_...
icon you find applicable
- for successfully accomplished tasks (
Example:
What did you do since yesterday?
- Accomplished the
Spotbugs java analyzer compareKey is not unique
https://gitlab.com/gitlab-org/gitlab-ee/issues/10860 - Still working on
Allow guests to create an issue from a vulnerability
https://gitlab.com/gitlab-org/gitlab-ee/issues/7813 - Catch-up on all emails and threads after the vacation
Slack Channels:
As our teams focus on different areas, we have Geekbot configured to broadcast to separate channels in addition to our common one at #s_secure-standup.
- Composition Analysis: #g_secure-composition-analysis-standup
- Static Analysis: #g_secure-static-analysis-standup
- Dynamic Analysis: #g_secure-dynamic-analysis-standup
- Frontend: #s_secure-frontend-standup
Recorded meetings
Our important meetings are recorded and published on YouTube, in the GitLab Secure Stage playlist. They give a good overview of the decision process, which is often a discussion with all the stakeholders. As we are a remote company, these video meetings help to synchronize and take decisions faster than commenting on issues. We prefer asynchronous work, but for large features and when the timing is tight, we can detail a lot of specifications. This will make the asynchronous work easier, since we have evaluated all edge cases.
Calendar
We welcome team members to join meetings that are on our shared calendar. The Secure Calendar is available to all logged in GitLab team members.
Staying informed
GitLab is an extremely active organization which generates a lot of news and activity each week. Everyone in Secure are encouraged to keep themselves informed as to what is happening in the larger organzation. Everyone is also encouraged to contribute to these channels and communication paradigms when you have information to share.
In addition to this, each group in Secure conducts a weekly synchronous meeting. These meetings are publicized on the Secure Calendar mentioned above. As always at GitLab, we strive to make meeting attendance optional.
Keeping others informed
In addition to keeping yourself informed, team members are encouraged to keep others informed as well. Secure groups have adopted a practice of including the following topics as standing agenda items in their weekly meetings, with example topics for each bullet point.
- Current status
- Work recently achieved against top priorities for that milestone.
- Pre-recorded demos are appreciated and encouraged as part of these updates.
- Newly discovered scope or dependencies.
- Work recently achieved against top priorities for that milestone.
- Risks
- Issues which are blocked or slowed, impacting whether they can be delivered in the desired timeframe.
- Help wanted
- Issues or topics on which the team or individuals on the team are getting stuck and could use some help.
- Praise
- Anyone doing a great job and you want to give them kudos?
- Any bit of work which has been delivered that’s exceptional?
Engineering Managers are responsible for populating this section of weekly group meetings, though everyone can contribute. In addition to helping the group keep itself informed about what’s happening each week, the SEM for Secure will collect this information weekly and broadcast a curated list to the section.
Technical onboarding
New hires should go through these steps and read the corresponding documentation when onboarding in the Secure Team. Every new hire will have an assigned onboarding issue that will guide them through the whole process.
Workflow and Refinement
Secure largely follows our Product Development Flow
See Issue Refinement to learn how we evaluate complexity, level of effort, our implementation plan and assign issue weights.
Shared pool of Frontend work
From time to time, the demand for frontend engineering tasks does not match the capacity available in Secure groups. If a group could use frontend support, they are encouraged to add the
~secure::frontend backlog
label to issues which are ready to be picked up. Additionally,
Engineering Managers are encouraged to look at this queue of work during milestone planning if there is frontend capacity available.
Coding standards and style guidelines
The Secure Team follows the coding standards and style guidelines outlined in the company-wide Contributor and Development Docs, however, please consult the following guidelines which are specific to the Secure Team:
Cross group collaboration
Some components of the architecture that support Secure features are shared between multiple groups like the common Go library, the Security Report Schemas, the rails parsers, etc.
Modifying these shared pieces might impact other groups so we should rely as much as possible on approval rules to ensure such changes are reviewed by the relevant teams before being merged.
Impactless two-way door changes could skip the approval process, please use sound judgement and common sense in such situations.
The author of changes should announce broadly the changes made on these components to raise awareness (weekly meeting agenda, slack channel).
Development of new analyzers
For a complete guide about developing a new analyzer please refer to our user documentation
Technical Documentation
As our product evolves, the engineering teams are researching ways to achieve new functionality and improve our architecture.
The outcome of this research can be found in our Technical Documentation section.
Data Sources
We keep a list of data sources in our internal wiki. This includes advisory databases, package license information, and related data.
Retrospectives
The Secure sub-department conducts retrospectives at the group level that follow our engineering workflow. Each group’s DRI is responsible to prepare and schedule the retrospective sync sessions and the async retrospective issues can be found in the corresponding project.
After all groups have completed their retrospective, we conduct a Section Retrospective.
NB: we use to have a sub-department wide retrospective whose issues are still accessible in the deprecated project.
Analytics
The Secure group reviews analytics to help understand customers and their usage of the tools. This data helps drive product and technical decisions. The following links show usage of Secure functionality.
Cross-functional Backlog
We also track our backlog of issues, including past due security and infradev issues, and total open SUS-impacting issues and bugs.
Merged Merge Request Types
MR Type labels help us report what we’re working on to industry analysts in a way that’s consistent across the engineering department. The dashboard below shows the trend of MR Types over time and a list of merged MRs.
Common Links
Dynamic Analysis Group
Secret Detection Group
Secure Engineering Refinement
Secure QA Process
Secure Technical Documentation
Secure, Composition Analysis
Secure, Vulnerability Research
Static Analysis Group
f94b0591
)