How to use GitLab security features to detect log4j vulnerabilities

Note: Out of an abundance of caution, we encourage users who are using older versions of GitLab SAST and Dependency Scanning to update to the latest versions. You can find more information and recommended actions in this blog post.

Any customer leveraging the recommended includes for GitLab SAST has automatically received the new patched versions released Dec 13, 2021.

In light of the recently discovered log4j vulnerabilities, we would like to demonstrate how GitLab can be used to assess and remediate the log4j vulnerability as well as other security vulnerabilities that may exist in your projects.

The solutions shared here are:

• Dependency Scanning (Ultimate)

• Container Scanning (Ultimate)

• Cluster image scanning (Ultimate)

• Advanced Search (Premium)

Free users wishing to access Premium and Ultimate features can do so by signing up for a free trial of GitLab.

Use GitLab dependency scanning to detect and mitigate log4j

vulnerabilities

Dependency scanning uses Gemnasium, which has been updated to detect the log4j vulnerability, to automatically find security vulnerabilities in your software dependencies.

Let’s try dependency scanning with a vulnerable project. Navigate to Create new project > Import project > from URL and use https://github.com/christophetd/log4shell-vulnerable-app.git.

Next, navigate to Security & Compliance > Security dashboard and select to configure Dependency Scanning. This will create a new merge request enabling the dependency scanner, and you can immediately see the first scanning results in the merge request.

Alternatively, you can edit the .gitlab-ci.yml configuration file and include the Dependency Scanning CI/CD template.

include:

- template: Security/Dependency-Scanning.gitlab-ci.yml

Create a new merge request and wait for the pipeline to finish. Inspect the security reports.

{: .shadow}

Take action on the critical vulnerability, open the details and create a new confidential security issue to follow-up.

{: .shadow}

After merging the MR to add dependency scanning, future MRs and code changes will detect the log4j vulnerabilities. This helps to avoid accidentally introducing older versions again. Open the Security report in Security & Compliance to get an overview of the vulnerabilities.

{: .shadow}

You can customize the default settings using CI/CD variables, for example increasing the log level to debug with SECURE_LOG_LEVEL: ‘debug’.

The project created in the examples above is located here.

Detect log4j vulnerabilities with Container Scanning

Vulnerabilities in container images can come not only from the source code for the application, but also from packages and libraries that are installed on the base image. Images can inherit packages and vulnerabilities from other container images using the FROM keyword in a Dockerfile. Container Scanning helps detect these vulnerabilities for the Operating System including packages. The latest release adds language vulnerability scans as a new optional feature to help detect the log4j library vulnerability using the underlying scanners (Trivy as default, Grype optional). You can also use this capability to scan remote images using the DOCKER_IMAGE variable.

You can enable the CS_DISABLE_LANGUAGE_VULNERABILITY_SCAN variable to scan for language specific packages. Please note that the additionally detected language dependencies can cause duplicates when you enable Dependency Scanning too.

To try it, navigate to CI/CD > Pipeline Editor and add the following configuration for Container Scanning:

include:
    - template: Security/Container-Scanning.gitlab-ci.yml

variables:
    # Use Trivy or Grype as security scanners (Trivy is the default in the included template)
    # CS_ANALYZER_IMAGE: "registry.gitlab.com/security-products/container-scanning/trivy:4"
    # CS_ANALYZER_IMAGE: "registry.gitlab.com/security-products/container-scanning/grype:4"
    # Detect language libraries as dependencies
    CS_DISABLE_LANGUAGE_VULNERABILITY_SCAN: "false"
    # Test the vulnerable log4j image 
    DOCKER_IMAGE: registry.gitlab.com/gitlab-de/playground/log4shell-vulnerable-app:latest 

Create a new branch, commit the changes and create a new MR. Once the pipeline has completed, inspect the security report in the MR.

{: .shadow}

After merging the MR, you can view the vulnerabilities that exist in your default branch by navigating to Security & Compliance > Vulnerability Report.

{: .shadow}

Inspect the vulnerability details to take action.

{: .shadow}

This feature is available for customers using the default CI/CD templates, or the tagged :4 scanner images from GitLab's Container Registry (registry.gitlab.com). If you are using custom images, please rebuild them based on the latest release.

Detect vulnerable containers in your Kubernetes cluster

You can use cluster image scanning in Kubernetes which uses Starboard and uses Trivy as a security scanner under the hood. Trivy’s vulnerability DB is able to detect CVE-2021-44228.

Let’s try it! A quick way to bring up a Kubernetes cluster is in Civo Cloud. Create an account, and follow the documentation on how to set up the CLI with an API token. Next, create a k3s cluster.

$ civo kubernetes create log4j

$ civo kubernetes config log4j --save

$ kubectl config use-context log4j

$ kubectl get node

registry.gitlab.com/gitlab-de/playground/log4shell-vulnerable-app:latest provides a vulnerable container image we can deploy and then scan.

$ vim deployment.yaml


apiVersion: apps/v1

kind: Deployment

metadata:
  name: log4j
spec:
  replicas: 2
  selector:
    matchLabels:
      app: log4j
  template:
    metadata:
      labels:
        app: log4j
    spec:
      containers:
        - image: registry.gitlab.com/gitlab-de/playground/log4shell-vulnerable-app:latest
          name: log4j

$ kubectl apply -f deployment.yaml

$ vim service.yaml


apiVersion: v1

kind: Service

metadata:
  name: log4j
  labels:
    app: log4j
spec:
  ports:
    - name: "log4j"
      port: 8080
  selector:
    app: log4j

$ kubectl apply -f service.yaml

Test the application container with port forwarding, and open your browser at http://localhost:80808. You can close the connection with ctrl+c.

$ kubectl port-forward service/log4j 8080:8080

After the deployment is finished, let’s add the cluster image scanning integration. Follow the Starboard Operator installation documentation. Next, configure the Kubernetes Cluster Image Scanning with GitLab.

The final step is to integrate the CI/CD template and run the pipelines.

include:
  - template: Security/Cluster-Image-Scanning.gitlab-ci.yml

Navigate into Security & Compliance > Vulnerability report and select the Operational vulnerabilities tab to inspect the vulnerabilities. There you can see that log4j was detected in the deployed application running in our Kubernetes cluster 💜.

{: .shadow}

Inspect the log4j vulnerability to see more details.

{: .shadow}

The full project is located here.

Search GitLab projects which use the log4j Java library

You can use the advanced search with scope blobs. Let’s try it! Navigate to your profile and add a new personal access token (PAT). Export it into the environment to access it in the next step:

$ export GITLAB_TOKEN=xxxxxxxxx


$ curl --header "PRIVATE-TOKEN: $GITLAB_TOKEN"
"https://gitlab.com/api/v4/search?scope=blobs&search=log4j" 

Tip: Install jq to format the JSON body. More insights in this blog post.

$ curl --header "PRIVATE-TOKEN: $GITLAB_TOKEN"
"https://gitlab.com/api/v4/search?scope=blobs&search=log4j" | jq

  {
    "basename": "src/main/resources/log4j",
    "data": "log4j.rootLogger=ERROR, stdout\n \n# Direct log messages to stdout\n",
    "path": "src/main/resources/log4j.properties",
    "filename": "src/main/resources/log4j.properties",
    "id": null,
    "ref": "9a1df407e1a5365950a77f715163f6dba915fdf4",
    "startline": 2,
    "project_id": 12345678
  },

You can use jq to further transform and filter the result set, for example only listing the paths where log4j as a string exists.

curl --header "PRIVATE-TOKEN: $GITLAB_TOKEN"
"https://gitlab.com/api/v4/search?scope=blobs&search=log4j" | jq -c '.[] |
select (.path | contains ("log4j"))' | jq

Next steps

The GitLab security team is continuing to proactively monitor the situation and ensure our product and customers are secure. We will continue to communicate should we identify additional opportunities to help our customers and community navigate through this situation. Please subscribe to our security alerts mailing list.

Please visit the public log4j-resources project and visit our forum for additional information.