Gitlab hero border pattern left svg Gitlab hero border pattern right svg

Vulnerability Management Overview

Vulnerability Management Overview

Vulnerability Management is the recurring process of identifying, classifying, prioritizing, mitigating, and remediating vulnerabilities. This overview will focus on infrastructure vulnerabilities and the operational vulnerability management process. This process is designed to provide insight into our environments, leverage GitLab for vulnerability workflows, promote healthy patch management among other preventative best-practices, and remediate risk; all with the end goal to better secure our environments.

To achieve these goals, we’ve partnered with Tenable and have deployed their software-as-a-service (SaaS) solution, Tenable.io, as our vulnerability scanner. Tenable.io allows us to focus on what is important; scanning for vulnerabilities, analyzing, and ingesting vulnerability data into GitLab as the starting point for our vulnerability management process.

The Vulnerability Management process

Arguably the most important step for a successful vulnerability management process is defining the scope that the process will cover. Security and Infrastructure partnered to come up with a scope that would make sure all of our critical environments and systems were covered during deployment. The following environments are currently in-scope for GitLab.com production:

Environment Project Production Deployed
GCP gitlab-production yes yes
GCP gitlab-ops yes yes
GCP gitlab-dr yes no
GCP gitlab-ci limited no
AWS gitlab.com yes no
Azure systegitlab yes no
Digital Ocean gitlab b.v. yes no

Note: If you believe a system you are responsible for should be included in the vulnerability management process, please contact Security Operations.

With these environments scoped out and Tenable scanners deployed, we can begin the vulnerability management process. Keep in mind that vulnerability management is a feedback loop - vulnerability scanners provide the vulnerability data which is analyzed and ingested to mitigate and remediate found vulnerabilities. Feedback from this process feeds into preventative initiatives that further secure our environments.

Currently, we break down vulnerability management into the following steps:

1. Vulnerability Scanning

This step is where we scan resources in our environments to identify vulnerabilities. Once setup, scans run on regular cadences that meet or exceed our compliance framework requirements.

2. Reporting/Analysis

Vulnerability scan data is exported and analyzed to provide consolidated vulnerability data we can ingest into GitLab.com for vulnerability remediation tracking. This is currently a manual process where we export vulnerability data into a spreadsheet and pull out pertinent information.

Tenable also provides reporting functionality that is used by our Compliance team to run reports for audits.

3. Ingestion

Once the data is prepared in a format that we can pull out the most important information, we can ingest into GitLab.com. Issues are opened in the Security Operations tracker to track the remediation process of the vulnerability. Another issue is opened in the Infrastructure issue tracker linking to the Security Operations issue; these are so that the work can properly get prioritized and scheduled according to Infrastructure team’s workflow.

Currently, we group vulnerabilities on a number of factors, such as severity and solution, as to consolidate the work required to remediate. If there are 10 unique vulnerabilities, but all require the same solution, it makes more sense to open a single issue to work through remediation.

Vulnerability issues should be tagged with the vulnerability type label. The following labels exist to track the vulnerability remediation workflow:

We also add the VM label to all Vulnerability issues to scope the issues in the Vulnerability Management issue board.

4. Validation

Validation is an important part of vulnerability management. This is where we investigate to ensure that the vulnerability being reported has properly been identified.

Vulnerabilities can sometimes be identified during a scan, but are not actually on the system. This can happen for a number of reasons, but most commonly is the result of misflagged ports or services. These are classified as false positives and would go through the process to be closed as a false positive.

5. Remediation

Remediation is the part of the process in which a validated vulnerability is fixed. The remediation process would be tracked in the corresponding vulnerability issue in the Security Operations issue tracker. SLAs are in place to help prioritize vulnerability based on severity. Once a vulnerability is remediated, we will run followup scans on the impacted systems to validate that the vulnerability is indeed remediated.

Vulnerability Issue Workflows

There are several ways a vulnerability issue can be closed - below are some common vulnerability workflows using the vulnerability labels as reference:

Closed as Remediated

The most common workflow is to close a vulnerability issue as Remediated. This means that a vulnerability has been validated and remediation has taken place. Below is the workflow:

graph TB SubGraph1 --> Node2 subgraph "Vulnerability Issue Status: Closed" Node2(Vulnerability:Remediated) end subgraph "Vulnerability Issue Status: Open" Node1[Vulnerability:Vulnerable] --> SubGraph1[Vulnerability:Validated] end
Closed as False Positive

A vulnerability must always be validated - but sometimes the validation can prove that a vulnerability is a false positive. Below is the workflow:

graph TB SubGraph1 --> Node2 subgraph "Vulnerability Issue Status: Closed" Node2(Vulnerability:FalsePositive) end subgraph "Vulnerability Issue Status: Open" Node1[Vulnerability:Vulnerable] --> SubGraph1[Vulnerability:Validated] end
Closed as Exception

Sometimes issues arise that would otherwise prevent a vulnerability from being remediated or mitigated. While commonly, these would result in an open Exception vulnerability issue status, there are unique cases where an issue can be closed as an exception. Below is the workflow:

graph TB SubGraph1 --> Node2 subgraph "Vulnerability Issue Status: Closed" Node2(Vulnerability:Exception) end subgraph "Vulnerability Issue Status: Open" Node1[Vulnerability:Vulnerable] --> SubGraph1[Vulnerability:Validated] end
Open as Exception

Closed issue via the Exception process are very rare. Generally, an exception is a non-permanent way to assume risk on a vulnerability due to extenuating circumstances in which remediation can not take place within the required SLAs. Below is the described the workflow:

graph TB SubGraph1 --> Node3 subgraph "Vulnerability Issue Status: Closed" Node3(Vulnerability:Remediated) end subgraph "Vulnerability Issue Status: Open" Node1[Vulnerability:Vulnerable] --> Node2(Vulnerability:Validated) --> SubGraph1[Vulnerability:Exception] end
Open as Mitigated

Another common workflow is when a vulnerability is validated and a fix is scheduled for some time in the future (within the SLA). If we're able to, we will put mitigation in place in the interim to reduce the risk from the vulnerability. Below is the described workflow:

graph TB SubGraph1 --> Node3 subgraph "Vulnerability Issue Status: Closed" Node3(Vulnerability:Remediated) end subgraph "Vulnerability Issue Status: Open" Node1[Vulnerability:Vulnerable] --> Node2(Vulnerability:Validated) --> SubGraph1[Vulnerability:Mitigated] end

6. Feedback

The last step is for Security Operations and Infrastructure to determine what we can learn from each vulnerability remediated. This may be an improvement on the vulnerability management process itself or establishing preventive mechanisms for a repetitive vulnerability type. This feedback will be documented in the vulnerability issue and could result in additional issues being opened.

As stated above, this process is a cyclical loop. Vulnerability scans are recurring, providing new vulnerability data that feed new vulnerability issues and update/escalate open issues.

Remediation SLAs

Security and Infrastructure have come up with remediated SLAs based on a multitude of factors, such as severity, scope, impact, etc. All of these factors will be considered when mapping the priority to GitLab’s priority labels. The SLAs are as follows:

Priority Severity Mapping Time to mitigate Time to remediate
S1/P1 Zero-day Within 24 hours As soon as possible
S2/P2 Critical N/A Within 30 days
S3/P3 High N/A Within 60 days

S1/P1 vulnerabilities would be worked on immediately through the incident management process and adhere to any timelines determined as such. This includes a 15-minute engagement and 24-hour mitigation SLA.

Note: Mitigation SLAs only apply to S1/P1 vulnerabilities

Vulnerability Scanning Schedule

Vulnerability scans occur on a weekly basis in our scoped environments. The schedule can be seen below:

The start times are always consistent - however, scan durations may fluctuate based on a multitude of factors. Generally, the production scans complete in under 2-hours. We’ve segmented the gitlab-production scans to reduce impact to the environment. We’ve also enabled load throttling, so if increased load is detected on the systems/networks being scanned, Tenable will reduce its footprint to further reduce impact.

The target groups used in these scans are setup using GCP VPC network ranges to ensure any newly provisioned resources are scanned without manually inputting the resources IP into the scan. We will leverage similar functionality during our AWS and Azure deployments.

Note: for more information, please visit the #tenable-notifications channel on Slack where there are links to documentation breaking down what hosts are in what scan group.

Contact

If you have any questions or concerns related to vulnerability management please contact Security Operations in #security-department channel on slack, by tagging @sec-ops-team in slack or @gitlab-com/gl-security/secops in a GitLab issue, or finally you can open an issue in the Security Operation issue tracker. All work being done to improve this process is also tracked in the issue tracker.

Any questions regarding ownership around vulnerability management can be answered in GitLab’s tech stack documentation.