The Composition Analysis group at GitLab is charged with developing solutions which perform Container Scanning, and Software Composition Analysis. See the exhaustive list of projects the group maintains.
The Composition Analysis group largely follows GitLab's Product Development Flow.
We leverage the issue's health status feature to communicate the progress of the issue.
All issues should be marked On Track at the beginning of a milestone. This is currently done manually by the Engineering Manager.
Raising risk early is important. The more time we have, the more options we have. As such, the team reviews issues every week and discusses items that Need Attention or are At Risk to possibly course correct and re-assign resources based on the team's priorities.
Follow these steps when raising or downgrading risk:
On Track - the work will be completed within the planned milestone.Needs Attention - the issue is blocked or has other factors that need to be discussed.At Risk - the issue is in jeopardy of missing the cutoff to ship within the planned milestone.In addition to the above workflow, the Composition Analysis group can be involved in some experiments, which might temporarily alter how we work.
There are no experiments in progress at the moment.
After the 19th, we conduct an asynchronous retrospective. You can find current and past retrospectives for Composition Analysis team in https://gitlab.com/gl-retrospectives/secure-sub-dept/composition-analysis.
We use a shared calendar to see when team members are off work: c_629844cc273be17e067767febe[email protected].
Access to this calendar is granted via this google group: [email protected].
To share your time-off:
Time Off by Deel application under the Apps menu.Home, click on Your Events to show a dropdown.Add calendar under Additional calendars to include?. Use the calendar ID above.To visualize the calendar:
Other calendars section in the left sidebar.Subscribe to calendar. Use the calendar ID above.On top of our development roadmap, engineering teams need to respond to additional requests (support, community contributions, security vulnerabilities). This increases context switching and depending on the load this can drastically impact our ability to achieve our plan. As a result, the Composition Analysis development team is actively reserving capacity each iteration to triage and address these requests. Each milestone, an engineer is designated to handle each one of these responsibilities, by following the schedule defined in this epic. The rotation follows the development cycle, which means from the 18th of current month to the 17th of the following month. The total time allocation represents around 10% of the engineering team.
Time allocation: 15% of 1 engineer.
Time allocation: 15% of 1 engineer.
#f_licese_database Slack channel for communication about these items, so other team members can provide the support.
#g_secure-composition-analysis-alerts for any incidents on the license-db infrastructure.
Time allocation: 15% of 1 engineer.
~priority::1. (See Bugs triaging process)~priority::1. (See Infradev triaging process)These items must be triaged continuously throughout the milestone which means they must be checked multiple times a week.
We are responsible for triaging vulnerabilities reported on 2 sets of projects: our analyzers and their upstream scanner software. For the latter, we've set up mirrors to run our security scans.
We use the security-triage-automation tool to handle the following tasks:
Note: Our goal is to migrate from security-triage-automation to Vulnmapper tool in the nearest future. However, so far we're following the existing process.
When using this tool to update vulnerabilities and issues, please keep track of the commands that were executed and add them to a private note in the reaction rotation issue.
Vulnerability Report Dashboards to verify that there are vulnerabilities that can be resolved.
No longer detected activity filter and apply the Confirmed and Needs triage status.No longer detected activity filter, and apply the No longer detected activity filter and apply the Confirmed and Needs triage status.security-triage-automation tool to resolve vulnerabilities and close their issues. This tool must be executed separately for each of the projects in the following categories:
Vulnerability Report Dashboards that vulnerabilities have been resolved.create FedRAMP vulnerability issues with CRITICAL, HIGH, MEDIUM, LOW, and UNKNOWN severity levels by executing the security-triage-automation tool to process vulnerabilities for a given project.This tool must be executed separately for each of the projects in the following categories:
All vulnerabilities must be remediated by their respective SLA. This process only describes how to generate the required FedRAMP vulnerability issues required for the report.
In cases where automation fails, you must create the deviation requests manually before the issues reach SLA. To do so, use the following procedure.
Vulnerability Details section with a link to the advisory (RedHat tracker usually), CVE ID, severity, and CVSS score.Justification Section with:The OS vendor has published an updated advisory for
, indicating that package has not yet had a fix released for this vulnerability. Until a fix is available for the package, this vulnerability cannot practically be remediated.
Attached Evidence section with:As this operational requirement represents a dependency on a vendor-published package to address this vulnerability, no additional evidence has been supplied. Please refer to the linked vendor advisory in the above justification.
/relate <issue_id>Update the security issue accordingly
/label ~"FedRAMP::Vulnerability" ~"FedRAMP::DR Status::Open"
/milestone %Backlog
We priortize findings by their CVSS severities and SLAs, and currently focus on security findings with these severity levels:
An exception is made for Container scanning findings - we focus only on findings with Critical severity.
Please utilize all the time you have set aside. If you complete all the ones at Critical and High, please continue to triage - we want to address all findings but we are working in a risk based order.
We use the Vulnerability Report with filters to focus on items matching our policy and reported on the relevant projects.
For each "Detected" item, investigate and either dismiss or confirm it. If it's not clear whether there's indeed a threat, escalate to our Application Security team.
For vulnerabilities discovered in upstream scanners, an issue must be created in GitLab's issue tracker, and we should work with the relevant Open Source community to help provide a resolution. As a last resort, we can patch locally or fork the upstream project temporarily to fix the vulnerability sooner.
Refer to Vulnerability status definitions in case you are unsure of what each of them mean.
When there is no doubt a vulnerability is a false-positive, it can be "Dismissed". Select the "Dismiss" option from the vulnerability status options. Finally, make sure to comment on the vulnerability status change notification to explain why.
Because of both the way severity is generically set in CVSS and automated scanners do not have all context for an application, many findings which may be high risk in other environments or scenarios are low risk for our users. The containers ingest code from a user project and that user has developer access, and the containers are ephemeral and related to a specific pipeline.
In some other cases, a finding is related to an upstream dependency or Operating System and there is no fix available and no fix planned. Please be sure to mark this issues using the labels; blocked or blocked upstream.
When an issue is both blocked for a few releases and low risk you may dismiss the finding with a note as to the reasoning. If there is an open issue notify the Application Security team with your specific reasoning and close the issue (if applicable). In the future we will specifically want to tag everything related to these findings as won't fix or blocked when they are being closed, for now that is only available on issues and not findings.
The following class of container scan vulnerabilities can be considered low risk:
To add items to the list above discuss repeatable finding patterns with Application Security, get approval from a leader in the security section, and add to this list.
If the vulnerability impacts a dependency:
For all other confirmed vulnerabilities, create a security issue to discuss and track the remediation.
When a vulnerability has been remediated, it can be "Resolved". When doing so, comment how it was remediated, then select the "Resolve" option from the vulnerability status options, and close the related vulnerability issue.
Unfortunately, creating a security issue can't be done yet via the "create issue" button from the vulnerability page or security dashboard as this only works when creating an issue in the same project where the error was reported and we've disabled the embedded issue tracker in our projects.
Instead, in our workflow we open all our issues in the main GitLab project.
As a workaround, you can copy and paste the content of the vulnerability page (this keeps markdown formatting!). Please also follow our Security guidelines about creating new security issues.
You can leverage quick actions to add the necessary labels.
/confidential
/label ~security ~"type::bug" ~"bug::vulnerability"
/label ~"section::sec" ~"devops::secure" ~"group::composition analysis"
<!-- depending on the affected project: -->
/label ~"Category:Software Composition Analysis"
/label ~"Category:Container Scanning"
It's important to add the ~security and ~"bug::vulnerability" labels as described above, because the AppSec Escalation Engine will automatically pick up any issues with these labels and add additional labels ~security-sp-label-missing and ~security-triage-appsec as well as mention the issue in the #sec-appsec Slack channel. At this point, the Stable Counterpart or Application Security team triage person will pick up the issue and assign a severity as part of the appsec triage rotation.
Once the issue is created, please add it to the vulnerability's linked items for ease of tracking.
Developers reporting the security issue should help the Application Security team assess the impact of the vulnerability, and update the issue description with an Impact section.
If immediate feedback is required, then add a comment to the vulnerability issue with an @-mention directed at one of the Security Engineers listed in the Stable Counterpart section, or ping them on slack.
For each open issue that has no Priority label ("Open" column), shortly investigate the bug (< 2h) and comment with your findings. Ideally you'd suggest Priority and Severity levels to guide PM decision. Depending on how confident you are, you can either set the labels by yourself, or make a suggestion in a comment, and ping PM.
Track how long you actually spent investigating each bug in the Composition Analysis Bug Triaging Time Tracker spreadsheet.
Please refer to our infradev process for more details.
If the image release process is failing, an incident should be created to track how it was detected, escalated, and resolved. Documenting our incidents makes it possible to search for previous incidents by keyword, labels, and other issue filters. We open all of our incidents in the main GitLab project.
Open a new incident and add a description of the problem along with any reproduction steps. Add the following labels so that we can track the incidents that have impacted composition analysis in the future.
<!--
Select one of the following severities
Ref: https://about.gitlab.com/handbook/engineering/quality/issue-triage/#severity
-->
/label ~"severity::1"
/severity S1
/label ~"severity::2"
/severity S2
/label ~"severity::3"
/severity S3
/label ~"severity::4"
/severity S4
<!--
Select one of the following priorities
Ref: https://about.gitlab.com/handbook/engineering/quality/issue-triage/#priority
-->
/label ~"priority::1"
/label ~"priority::2"
/label ~"priority::3"
/label ~"priority::4"
/label ~"section::sec" ~"devops::secure" ~"group::composition analysis" ~"type::bug" ~"bug::availability"
<!--
Select one of the following categories
-->
/label ~"Category:Dependency Scanning"
/label ~"Category:Container Scanning"
/label ~"Category:License Compliance"
To help our Product Manager prioritize maintenance issues, the engineering team assigns them a priority label.
~maintenance::refactor).Upon joining Composition Analysis group, team members are suppose to become either reviewers or maintainers for all projects maintained by the group. The process how to become maintainer is described in the general Code review guidelines.
The Composition Analysis group maintains several projects to provide our scanning capabilities.
Additional notes:
As some of our analyzers rely on open source software, we include them in our security testing to increase coverage and reduce risk.
To do so, we mirror their repository and execute our security scans on them:
The vulnerabilities reported on the currently used version of the scanner are automatically reported in the group level Vulnerability Report and triaged as part of our security vulnerabilities triaging process.
SCANNER_VERSION variable in the analyzer's Dockerfile). Use exact commit if there is no git tag for the corresponding release we use.VERSION-security-checks where VERSION is the version of the upstream scanner we currently use (e.g. v6.12.0)..gitlab-ci.yml configuration file to configure all compatible security scans.VERSION-security-checks the default branch, so that reported vulnerabilities are showing up on the dashboards and vulnerability reports.We check for new releases of the upstream scanners on a monthly basis, as part of our release issue. When an update is available, a new issue is created using the update scanner issue template and added to the next milestone.
Every analyzer relying on an upstream scanner has a "How to update the upstream Scanner" section in their readme detailing the process. This includes a verification for possible new security vulnerabilities and a license check which are detailed below.
Before releasing an analyzer with a newer version of its upstream scanner, we must ensure it is exempt of security vulnerabilities matching our current policy.
NEW_VERSION-security-checks..gitlab-ci.yml configuration file from the current VERSION-security-check branch.NEW_VERSION-security-checks and proceed with the update of the analyzer to use this newer version.Before releasing an analyzer with a newer version of its upstream scanner, we must ensure its license has not changed or is still compatible with our policy.
(Sisense↗) We also track our backlog of issues, including past due security and infradev issues, and total open System Usability Scale (SUS) impacting issues and bugs.
(Sisense↗) MR Type labels help us report what we're working on to industry analysts in a way that's consistent across the engineering department. The dashboard below shows the trend of MR Types over time and a list of merged MRs.
(Sisense↗) Flaky test are problematic for many reasons.
(Sisense↗) Slow tests are impacting the GitLab pipeline duration.