How to benchmark security tools: a case study using WebGoat

As your organization grows, the necessity for having automated security tools be a component of your development pipeline will increase. According to the latest BSIMM10 study, full-time security members represented just 1.37% of the number of developers. That's one security team member for every 73 developers. Without automated tooling assisting security teams, vulnerabilities may more easily find their way into production.

When tasked to compare security tools, having the knowledge to judge a fair benchmark is paramount for success. We're going to take a very in-depth look at WebGoat's lessons in this blog post.

Running a fair benchmark

There are many factors that need to be taken into consideration when comparing various security tools.

Is the tool I am testing right for my organization?
Do the applications I chose to run the tools against reflect what my organization uses?
Do the applications contain real security issues in real world scenarios, or are they synthetic?
Are the results consistent across runs? Are they actionable?

Choosing the right tool

Some tools are developer focused, while others may be tailored to security teams. Highly technical tools built for security teams may give better results, but if it requires domain expertise and significant tuning, it may end up being a worse choice for your organization.

Your organization may also be more concerned about a tool which can run relatively quickly within your development pipeline. If your developers need results immediately, a SAST or DAST tool which takes hours or days to complete may be next to worthless.

Choosing applications

It's important when comparing tools that they are run against applications that have been developed in-house or closely mirror what your development teams are creating. A SAST tool that's excellent at finding Java security issues will not necessarily translate to one that's great at finding actionable issues in a NodeJS application. If your organization has a diverse set of languages you should run each tool against applications that reflect those environments.

Real world flaws

Applications like WebGoat or OWASP's Java Benchmark do not represent real world applications. Most vulnerabilities have been purposely injected into very simple data and code flows. The majority of flaws in WebGoat exist in the same Java class where the source of user input is defined. In reality, a large number of security issues will be hidden in nested layers of abstraction or multiple function or method calls. You'll want to ensure your test suite of applications includes real world applications and the tools can traverse these complex flows to find potential flaws.

Even if a tool is excellent at finding language specific issues, it may or may not support the development team's choice in frameworks. Most SAST tools need to add support for specific frameworks. If the tool supports the language of choice but does not support the particular framework, there will be a higher chance for poor results.

Are results consistent and useful?

Analysis of applications should be run multiple times – DAST tools in particular have a difficult time with consistency due to the dynamic nature of testing live applications. Ensure each tool is run the same number of times to gather enough data points to make a clear decision.

Security tools tend to over report issues and this can end up causing alert fatigue and reduce the value of the tool. It's important to tune the tools to reduce the number of non-actionable results. Just pay attention to how much time is required to maintain this tuning effort and be sure to include this in the final decision.

WebGoat as a benchmarking target

WebGoat is a known vulnerable application that was built to help developers and people interested in web application security understand various flaws and risks to applications. Over the years it has seen numerous contributions to the lessons and became a de facto standard for learning about web application security.

Due to the increase in popularity of this project, customers have chosen to rely on using it as a benchmark when assessing the capabilities of various SAST and DAST tools. The purpose of this post is to outline some potential pitfalls when using WebGoat as a target for analysis.

WebGoat was built as a learning aid, not for benchmarking purposes. Certain methods chosen to demonstrate vulnerability do not actually result in real flaws being implemented in WebGoat. To a human attempting to exercise certain flaws, it may look like they've succeeded in finding an issue. In reality, the WebGoat application just makes it appear like they've discovered a flaw.

This can cause confusion when an automated tool is run against WebGoat. To a human, they expect the tool to find a flaw at a particular location. Since a number of lessons include synthetic flaws, the tools will fail to report them.

For this post, GitLab's Vulnerability Research Team used the v8.1.0 release of WebGoat, however a number of the issues identified will be applicable to older versions as well.

WebGoat's architecture

The focus of this post is on WebGoat and in particular as a target for benchmarking. The WebGoat repository has grown in size over the years and now includes multiple sub-projects:

webgoat-container - This project holds the static content as well as the Spring Boot Framework's lesson scaffolding. The frontend is built using Backbone.js.
webgoat-images - Contains a Vagrant file for training purposes.
webgoat-integration-tests - Contains test files
webgoat-lessons - Contains the source and resources for the lessons.
webgoat-server - The primary entry point for the SpringBootApplication.
webwolf - An entirely separate project for assisting users in exploiting various lessons.

When building the WebGoat target application, the webgoat-container, webgoat-server and webgoat-lessons are all included into a single SpringBoot server packaged as a JAR artifact.

For the most part, lessons that contain legitimate flaws exist in a single class file. Certain SAST tools which implement extensive taint tracking capabilities may not be fully exercised. The end result being, SAST tools with limited capabilities will find just as many flaws as more advanced tools that can handle intra-procedural taint flows. It would be better to benchmark these tools against relatively complex applications versus comparing them with WebGoat's simplistic flaws.

Analysis methodology

Only the webgoat-container, webgoat-server and webgoat-lessons projects are included in our analysis of WebGoat for SAST/DAST tools. The other projects webgoat-images, webgoat-integration-tests and webwolf are not included.

Our analysis follows the lessons and attempts to identify in the source where the flaws, if any, exist. If a lesson is a description or does not process user input, it is not included in the flaw category listing below.

Each lesson is broken down to cover the following:

Flaw category
Lesson with title
Link as viewable from a browser
Source
Lesson's purpose
Relevant source code
Whether DAST/SAST would be able to find and the probability level
- Possible: A DAST/SAST tool should be able to find the flaw with little to no configuration changes or custom settings
- Probable: A DAST/SAST tool could find the flaw with some configuration changes or custom settings
- Improbable: A DAST/SAST tool most likely will not be able to find the flaw, due to either hardcoded values or other conditions that are not realistic
- Impossible: A DAST/SAST tool would not be able to find any flaw due to it being completely synthetic or other unrealistic circumstances
Reasoning why it can or can't be found with a SAST or DAST tool
Example attack where applicable.

In many places there are additional flaws that exist in the code but are not part of the lesson; we will highlight some of these but not exhaustively.

Please note this post contains spoilers, if you are new to WebGoat as a learning tool and wish to use it for study, it is recommended to do that first before reading our analysis.

(A1) Injection

Findings

(A1) Injection > SQL Injection (intro) > What is SQL?

Link: http://localhost:8080/WebGoat/start.mvc#lesson/SqlInjection.lesson/1

Source:

webgoat-lessons/sql-injection/src/main/java/org/owasp/webgoat/sql_injection/introduction/SqlInjectionLesson2.java

Lesson:

This lesson is for practicing raw SQL queries – it allows anyone to run full SQL queries without parameterized statements.

webgoat-lessons/sql-injection/src/main/java/org/owasp/webgoat/sql_injection/introduction/SqlInjectionLesson2.java#L55-65 

public AttackResult completed(@RequestParam String query) {
    return injectableQuery(query);
}

protected AttackResult injectableQuery(String query) {
    try (var connection = dataSource.getConnection()) {
        Statement statement = connection.createStatement(TYPE_SCROLL_INSENSITIVE, CONCUR_READ_ONLY);
        ResultSet results = statement.executeQuery(query);
        StringBuffer output = new StringBuffer();

        results.first();
        ...

Can SAST Find?

Possible

Can DAST Find?

Probable

SAST Reasoning:

It should be relatively straightforward for a SAST tool to identify that the query comes from a known source (line 55) and is used in a statement's executeQuery method.

DAST Reasoning:

While it should not be difficult for a DAST tool to find such a vulnerability, most DAST tools are not built with attack strings that attempt direct SQL queries. DAST SQL Injection tests are almost always trying to inject into the middle of an already existing SQL query. There is a good chance most DAST tools will not find the /SqlInjection/attack2 endpoint's query parameter vulnerable.

Example Attack:

query=(SELECT repeat('a',50000000) from information_schema.tables) will take ~3 seconds to complete.

(A1) Injection > SQL Injection (intro) > Data Manipulation Language (DML)

Link: http://localhost:8080/WebGoat/start.mvc#lesson/SqlInjection.lesson/2

Source:

webgoat-lessons/sql-injection/src/main/java/org/owasp/webgoat/sql_injection/introduction/SqlInjectionLesson3.java

Lesson:

This lesson is for practicing raw SQL queries – it allows anyone to run UPDATE/INSERT SQL queries without parameterized statements.

webgoat-lessons/sql-injection/src/main/java/org/owasp/webgoat/sql_injection/introduction/SqlInjectionLesson3.java#L56-65

public AttackResult completed(@RequestParam String query) {
    return injectableQuery(query);
}

protected AttackResult injectableQuery(String query) {
    try (Connection connection = dataSource.getConnection()) {
        try (Statement statement = connection.createStatement(TYPE_SCROLL_INSENSITIVE, CONCUR_READ_ONLY)) {
            Statement checkStatement = connection.createStatement(TYPE_SCROLL_INSENSITIVE,
                    CONCUR_READ_ONLY);
            statement.executeUpdate(query);
        ...

Can SAST Find?

Possible

Can DAST Find?

Probable

SAST Reasoning:

It should be relatively straightforward for a SAST tool to identify that the query comes from a known source on line 56 and is used in a statement's executeUpdate method.

DAST Reasoning:

Example Attack:

query=insert into employees (first_name) (SELECT repeat('a', 50000000) from employees) will take ~2 seconds to error out (where repeat('a', 5) takes 200ms).

(A1) Injection > SQL Injection (intro) > Data Definition Language (DDL)

Link: http://localhost:8080/WebGoat/start.mvc#lesson/SqlInjection.lesson/3

Source:

webgoat-lessons/sql-injection/src/main/java/org/owasp/webgoat/sql_injection/introduction/SqlInjectionLesson4.java

Lesson:

This lesson is for practicing raw SQL queries – it allows anyone to create tables via SQL queries without parameterized statements.

webgoat-lessons/sql-injection/src/main/java/org/owasp/webgoat/sql_injection/introduction/SqlInjectionLesson4.java#L52-59

public AttackResult completed(@RequestParam String query) {
    return injectableQuery(query);
}

protected AttackResult injectableQuery(String query) {
    try (Connection connection = dataSource.getConnection()) {
        try (Statement statement = connection.createStatement(TYPE_SCROLL_INSENSITIVE, CONCUR_READ_ONLY)) {
            statement.executeUpdate(query);
        ...

Can SAST Find?

Possible

Can DAST Find?

Probable

SAST Reasoning:

It should be relatively straightforward for a SAST tool to identify that the query comes from a known source on line 53 and is used in a statement's executeUpdate method.

DAST Reasoning:

While it should not be difficult for a DAST tool to find such a vulnerability, most DAST tools are not built with attack strings that attempt direct SQL injection queries. DAST SQL Injection tests are almost always trying to inject into the middle of an already existing SQL query. There is a good chance most DAST tools will not find the /SqlInjection/attack4 endpoint's query parameter vulnerable to sql injection.

Example Attack:

query=insert into employees (first_name) (SELECT repeat('a', 50000000) from information_schema.tables) will take ~2 seconds to error out (where repeat('a', 5) takes 200ms).

(A1) Injection > SQL Injection (intro) > Data Control Language (DCL)

Link: http://localhost:8080/WebGoat/start.mvc#lesson/SqlInjection.lesson/4

Source:

webgoat-lessons/sql-injection/src/main/java/org/owasp/webgoat/sql_injection/introduction/SqlInjectionLesson5.java

Lesson:

This lesson is for practicing raw SQL queries – it pretends to allow users to run grant/alter on tables via SQL queries. However, it is not calling any SQL functionality.

webgoat-lessons/sql-injection/src/main/java/org/owasp/webgoat/sql_injection/introduction/SqlInjectionLesson5.java#46-51

String regex = "(?i)^(grant alter table to [']?unauthorizedUser[']?)(?:[;]?)$";
StringBuffer output = new StringBuffer();

// user completes lesson if the query is correct
if (query.matches(regex)) {
    output.append("<span class='feedback-positive'>" + query + "</span>");

Can SAST Find?

Impossible

Can DAST Find?

Impossible

SAST Reasoning:

This is a synthetic vulnerability that does not actually call any database functionality.

DAST Reasoning:

This is a synthetic vulnerability that does not actually call any database functionality.

(A1) Injection > SQL Injection (intro) > What is SQL injection?

Link: http://localhost:8080/WebGoat/start.mvc#lesson/SqlInjection.lesson/5

Source:

Static content only

Lesson:

This lesson is for practicing SQL Injection query string generation, but does not call any server side functionality.

Can SAST Find?

Impossible

Can DAST Find?

Impossible

SAST Reasoning:

This lesson does not actually call any server side functionality.

DAST Reasoning:

This lesson does not actually call any server side functionality.

(A1) Injection > SQL Injection (intro) > Try It! String SQL injection

Link: http://localhost:8080/WebGoat/start.mvc#lesson/SqlInjection.lesson/8

Source:

webgoat-lessons/sql-injection/src/main/java/org/owasp/webgoat/sql_injection/introduction/SqlInjectionLesson5a.java

Lesson:

This lesson provides a form for testing out SQL Injection against database functionality with the goal of returning all results from a table.

webgoat-lessons/sql-injection/src/main/java/org/owasp/webgoat/sql_injection/introduction/SqlInjectionLesson5a.java#L53-L54

public AttackResult completed(@RequestParam String account, @RequestParam String operator, @RequestParam String injection) {
    return injectableQuery(account + " " + operator + " " + injection);
...

webgoat-lessons/sql-injection/src/main/java/org/owasp/webgoat/sql_injection/introduction/SqlInjectionLesson5a.java#L57-62

protected AttackResult injectableQuery(String accountName) {
    String query = "";
    try (Connection connection = dataSource.getConnection()) {
        query = "SELECT * FROM user_data WHERE first_name = 'John' and last_name = '" + accountName + "'";
        try (Statement statement = connection.createStatement(ResultSet.TYPE_SCROLL_INSENSITIVE, ResultSet.CONCUR_UPDATABLE)) {
            ResultSet results = statement.executeQuery(query);
    ...

Can SAST Find?

Possible

Can DAST Find?

Possible

SAST Reasoning:

The executeQuery method is called from a dynamically concatenated string containing user input.

DAST Reasoning:

The three inputs account, operator and injection are all concatenated together and each parameter could technically be used as an attack vector. In this case a DAST tool will most likely suffer from over reporting duplicate flaws, as the three input vectors all end up being used in the same resultant query string.

Example Attack(s):

account=Smith' or 1%3d1--&operator=&injection=
account=&operator=Smith' or 1%3d1--&injection=
account=&operator=&injection=Smith' or 1%3d1--

(A1) Injection > SQL Injection (intro) > Try It! Numeric SQL injection

Link: http://localhost:8080/WebGoat/start.mvc#lesson/SqlInjection.lesson/9

Source:

webgoat-lessons/sql-injection/src/main/java/org/owasp/webgoat/sql_injection/introduction/SqlInjectionLesson5b.java

Lesson:

This lesson provides a form for testing out SQL Injection when the column type is restricted to numerical values. The goal is to return all results from a table. While the resultant query does make use of prepared statements, it incorrectly concatenates user input for the userid column.

webgoat-lessons/sql-injection/src/main/java/org/owasp/webgoat/sql_injection/introduction/SqlInjectionLesson5b.java#L56-71

String queryString = "SELECT * From user_data WHERE Login_Count = ? and userid= " + accountName;
try (Connection connection = dataSource.getConnection()) {
        PreparedStatement query = connection.prepareStatement(queryString, ResultSet.TYPE_SCROLL_INSENSITIVE, ResultSet.CONCUR_READ_ONLY);
        ...
        ResultSet results = query.executeQuery();
        ...

Can SAST Find?

Possible

Can DAST Find?

Possible

SAST Reasoning:

While the query is correctly used in a PreparedStatement only the Login_Count is parameterized, the accountName is still dynamically assigned, leading to SQL Injection.

DAST Reasoning:

DAST tools should attempt to inject into both parameters, and provided the correct attack string is used, should successfully identify that the userid parameter is vulnerable to SQL Injection.

Example Attack:

login_count=1&userid=1+or+1%3D1

(A1) Injection > SQL Injection (intro) > Compromising confidentiality with String SQL injection

Link: http://localhost:8080/WebGoat/start.mvc#lesson/SqlInjection.lesson/10

Source:

webgoat-lessons/sql-injection/src/main/java/org/owasp/webgoat/sql_injection/introduction/SqlInjectionLesson8.java

Lesson:

This lesson provides a form for testing out SQL Injection where the goal is to return all results from a table.

webgoat-lessons/sql-injection/src/main/java/org/owasp/webgoat/sql_injection/introduction/SqlInjectionLesson8.java#L59-65

String query = "SELECT * FROM employees WHERE last_name = '" + name + "' AND auth_tan = '" + auth_tan + "'";

try (Connection connection = dataSource.getConnection()) {
    try {
        Statement statement = connection.createStatement(ResultSet.TYPE_SCROLL_INSENSITIVE, ResultSet.CONCUR_UPDATABLE);
        log(connection, query);
        ResultSet results = statement.executeQuery(query);
...

Can SAST Find?

Possible

Can DAST Find?

Possible

SAST Reasoning:

The name and auth_tan user supplied values are dynamically assigned to a query string, leading to SQL Injection.

DAST Reasoning:

DAST tools should attempt to inject into both parameters which are valid attack vectors leading to exploitable SQL Injection.

Example Attack:

name=&auth_tan=1'+or+1%3D1--
name=1'+or+1%3D1--&auth_tan=

(A1) Injection > SQL Injection (intro) > Compromising Integrity with Query chaining

Link: http://localhost:8080/WebGoat/start.mvc#lesson/SqlInjection.lesson/11

Source:

webgoat-lessons/sql-injection/src/main/java/org/owasp/webgoat/sql_injection/introduction/SqlInjectionLesson9.java

Lesson:

This lesson provides a form for testing out SQL Injection where the goal is to modify a users salary.

webgoat-lessons/sql-injection/src/main/java/org/owasp/webgoat/sql_injection/introduction/SqlInjectionLesson9.java#L61-66

String query = "SELECT * FROM employees WHERE last_name = '" + name + "' AND auth_tan = '" + auth_tan + "'";
try (Connection connection = dataSource.getConnection()) {
    try {
        Statement statement = connection.createStatement(TYPE_SCROLL_SENSITIVE, CONCUR_UPDATABLE);
        SqlInjectionLesson8.log(connection, query);
        ResultSet results = statement.executeQuery(query);
...

Can SAST Find?

Possible

Can DAST Find?

Possible

SAST Reasoning:

The name and auth_tan user supplied values are dynamically assigned to a query string, leading to SQL Injection.

DAST Reasoning:

DAST tools should attempt to inject into both parameters name and auth_tan which are valid attack vectors leading to exploitable SQL Injection.

Example Attack:

name=&auth_tan=1'+or+1%3D1--
name=1'+or+1%3D1--&auth_tan=

(A1) Injection > SQL Injection (intro) > Compromising Availability

Link: http://localhost:8080/WebGoat/start.mvc#lesson/SqlInjection.lesson/12

Source:

webgoat-lessons/sql-injection/src/main/java/org/owasp/webgoat/sql_injection/introduction/SqlInjectionLesson10.java

Lesson:

This lesson provides a form for testing SQL Injection where the goal is remove evidence of attacks.

webgoat-lessons/sql-injection/src/main/java/org/owasp/webgoat/sql_injection/introduction/SqlInjectionLesson10.java#L58-63

String query = "SELECT * FROM access_log WHERE action LIKE '%" + action + "%'";

        try (Connection connection = dataSource.getConnection()) {
            try {
                Statement statement = connection.createStatement(ResultSet.TYPE_SCROLL_INSENSITIVE, ResultSet.CONCUR_READ_ONLY);
                ResultSet results = statement.executeQuery(query);
...

Can SAST Find?

Possible

Can DAST Find?

Possible

SAST Reasoning:

The action user supplied values are dynamically concatenated to a query string, leading to SQL Injection.

DAST Reasoning:

DAST tools should attempt to inject into the action_string parameter, leading to exploitable SQL Injection.

Example Attack:

action_string=1'+or+1%3D1--

(A1) Injection > SQL Injection (advanced) > Try It! Pulling data from other tables

Link: http://localhost:8080/WebGoat/start.mvc#lesson/SqlInjectionAdvanced.lesson/2

Source:

webgoat-lessons/sql-injection/src/main/java/org/owasp/webgoat/sql_injection/advanced/SqlInjectionLesson6a.java

Lesson:

This lesson is to demonstrate how to extract data from tables other than the one the query was defined to execute against.

webgoat-lessons/sql-injection/src/main/java/org/owasp/webgoat/sql_injection/advanced/SqlInjectionLesson6a.java#L60-67

query = "SELECT * FROM user_data WHERE last_name = '" + accountName + "'";
//Check if Union is used
if (!accountName.matches("(?i)(^[^-/*;)]*)(\\s*)UNION(.*$)")) {
    usedUnion = false;
}
try (Statement statement = connection.createStatement(ResultSet.TYPE_SCROLL_INSENSITIVE,
        ResultSet.CONCUR_READ_ONLY)) {
    ResultSet results = statement.executeQuery(query);
...

Can SAST Find?

Possible

Can DAST Find?

Possible

SAST Reasoning:

The accountName user supplied value is dynamically assigned to a query string, leading to SQL Injection.

DAST Reasoning:

DAST tools should attempt to inject into the userid_6a parameter, leading to exploitable SQL Injection.

Example Attack:

userid_6a=1'+or+1%3D1--

(A1) Injection > SQL Injection (advanced) > (no title)

Link: http://localhost:8080/WebGoat/start.mvc#lesson/SqlInjectionAdvanced.lesson/4

Source:

webgoat-lessons/sql-injection/src/main/java/org/owasp/webgoat/sql_injection/advanced/SqlInjectionChallenge.java

Lesson:

This lesson is a challenge to attempt to extract data from the database that leads to the attacker being able to login as a different user by executing SQL Injection attacks.

webgoat-lessons/sql-injection/src/main/java/org/owasp/webgoat/sql_injection/advanced/SqlInjectionChallenge.java#L63-65

String checkUserQuery = "select userid from sql_challenge_users where userid = '" + username_reg + "'";
Statement statement = connection.createStatement();
ResultSet resultSet = statement.executeQuery(checkUserQuery);
...

Can SAST Find?

Possible

Can DAST Find?

Possible

SAST Reasoning:

The username_reg user supplied value is dynamically assigned to a query string, leading to SQL Injection.

DAST Reasoning:

DAST tools should attempt to inject into the username_reg parameter with a timing or blind sql injection based attack string, leading to exploitable SQL Injection.

Example Attack:

username_reg='%2b(select+repeat('a', 50000000)+from+information_schema.tables)%2b'&email_reg=asdf&password_reg=asdf&confirm_password_reg=asdf

(A1) Injection > SQL Injection (mitigation) > Input validation alone is not enough!! (Lesson 9)

Link: http://localhost:8080/WebGoat/start.mvc#lesson/SqlInjectionMitigations.lesson/8

Source:

webgoat-lessons/sql-injection/src/main/java/org/owasp/webgoat/sql_injection/mitigation/SqlOnlyInputValidation.java
webgoat-lessons/sql-injection/src/main/java/org/owasp/webgoat/sql_injection/advanced/SqlInjectionLesson6a.java

Lesson:

This lesson demonstrates filtering of user input not being sufficient for protecting against SQL injection attacks. This particular case looks to see if the input string contains a space and returns an error if it does.

webgoat-lessons/sql-injection/src/main/java/org/owasp/webgoat/sql_injection/mitigation/SqlOnlyInputValidation.java#L48-L52

public AttackResult attack(@RequestParam("userid_sql_only_input_validation") String userId) {
if (userId.contains(" ")) {
    return failed(this).feedback("SqlOnlyInputValidation-failed").build();
}
AttackResult attackResult = lesson6a.injectableQuery(userId);
...

webgoat-lessons/sql-injection/src/main/java/org/owasp/webgoat/sql_injection/advanced/SqlInjectionLesson6a.java#56-67

public AttackResult injectableQuery(String accountName) {
    String query = "";
    try (Connection connection = dataSource.getConnection()) {
        boolean usedUnion = true;
        query = "SELECT * FROM user_data WHERE last_name = '" + accountName + "'";
        //Check if Union is used
        if (!accountName.matches("(?i)(^[^-/*;)]*)(\\s*)UNION(.*$)")) {
            usedUnion = false;
        }
        try (Statement statement = connection.createStatement(ResultSet.TYPE_SCROLL_INSENSITIVE,
                ResultSet.CONCUR_READ_ONLY)) {
            ResultSet results = statement.executeQuery(query);

Can SAST Find?

Possible

Can DAST Find?

Possible

SAST Reasoning:

SAST tools would most likely be flag this as a flaw existing under the webgoat-lessons/sql-injection/src/main/java/org/owasp/webgoat/sql_injection/advanced/SqlInjectionLesson6a.java class. The SqlOnlyInputValidation.java file is calling the same method, but the vulnerability exists in SqlInjectionLesson6a.java. Where the flaw is reported depends on how the SAST tool was designed. For example if the SAST tool is capable of tracking taint across intra-procedural calls it may flag the call to lesson6a.injectableQuery(userId) on line 52. If the SAST tool was designed to only parse the Abstract Syntax Tree (AST) it may only report the flaw in SqlInjectionLesson6a.java on line 67.

DAST Reasoning:

Any input filtering done on potential attack vectors makes it more difficult for DAST tools to identify exploitable issues. There is a chance some DAST tools would not find the userid_sql_only_input_validation parameter vulnerable to SQL Injection.

Example Attack:

userid_sql_only_input_validation='%2b(SELECT/**/repeat(char(60),50000000)from/**/information_schema.tables)%2b'

(A1) Injection > SQL Injection (mitigation) > Input validation alone is not enough!! (Lesson 10)

Link: http://localhost:8080/WebGoat/start.mvc#lesson/SqlInjectionMitigations.lesson/9

Source:

webgoat-lessons/sql-injection/src/main/java/org/owasp/webgoat/sql_injection/mitigation/SqlOnlyInputValidationOnKeywords.java
webgoat-lessons/sql-injection/src/main/java/org/owasp/webgoat/sql_injection/advanced/SqlInjectionLesson6a.java

Lesson:

This lesson demonstrates filtering of user input not being sufficient for protecting against SQL Injection attacks. This particular case looks to see if the input string contains SELECT or FROM keywords and replaces them with empty strings, it additionally checks if the input contains a space and returns an error if it does.

webgoat-lessons/sql-injection/src/main/java/org/owasp/webgoat/sql_injection/mitigation/SqlOnlyInputValidationOnKeywords.java#L48-L53

public AttackResult attack(@RequestParam("userid_sql_only_input_validation_on_keywords") String userId) {
    userId = userId.toUpperCase().replace("FROM", "").replace("SELECT", "");
    if (userId.contains(" ")) {
        return failed(this).feedback("SqlOnlyInputValidationOnKeywords-failed").build();
    }
    AttackResult attackResult = lesson6a.injectableQuery(userId);
    ...

webgoat-lessons/sql-injection/src/main/java/org/owasp/webgoat/sql_injection/advanced/SqlInjectionLesson6a.java#56-67

public AttackResult injectableQuery(String accountName) {
    String query = "";
    try (Connection connection = dataSource.getConnection()) {
        boolean usedUnion = true;
        query = "SELECT * FROM user_data WHERE last_name = '" + accountName + "'";
        //Check if Union is used
        if (!accountName.matches("(?i)(^[^-/*;)]*)(\\s*)UNION(.*$)")) {
            usedUnion = false;
        }
        try (Statement statement = connection.createStatement(ResultSet.TYPE_SCROLL_INSENSITIVE,
                ResultSet.CONCUR_READ_ONLY)) {
            ResultSet results = statement.executeQuery(query);

Can SAST Find?

Possible

Can DAST Find?

Possible

SAST Reasoning:

SAST tools would most likely flag this as a flaw existing under the webgoat-lessons/sql-injection/src/main/java/org/owasp/webgoat/sql_injection/advanced/SqlInjectionLesson6a.java class. The SqlOnlyInputValidationOnKeywords.java file is calling the same method, but the vulnerability exists in SqlInjectionLesson6a.java. Where the flaw is reported depends on how the SAST tool was designed. For example if the SAST tool is capable of tracking taint across intra-procedural calls it may flag the call to lesson6a.injectableQuery(userId) on line 53. If the SAST tool was designed to only parse the Abstract Syntax Tree (AST) it may only report the flaw in SqlInjectionLesson6a.java on line 67.

DAST Reasoning:

Any input filtering done on potential attack vectors makes it much more difficult for DAST tools to identify exploitable issues. There is a good chance most DAST tools would not find the userid_sql_only_input_validation_on_keywords parameter vulnerable.

Example Attack:

userid_sql_only_input_validation_on_keywords='%2b(SELselectECT/**/repeat(char(60),50000000)FRfromOM/**/information_schema.tables)%2b'

(A1) Injection > SQL Injection (mitigation) > (no title) (Lesson 12)

Link: http://localhost:8080/WebGoat/start.mvc#lesson/SqlInjectionMitigations.lesson/11

Source:

webgoat-lessons/sql-injection/src/main/java/org/owasp/webgoat/sql_injection/mitigation/Servers.java

Lesson:

This lesson includes a SQL Injection vulnerability in a column field of a SQL query, which can not be specified by a parameter in a parameterized query. As such, it uses a dynamically generated query string with user input for specifying the column name.

webgoat-lessons/sql-injection/src/main/java/org/owasp/webgoat/sql_injection/mitigation/Servers.java#L69-L74

 public List<Server> sort(@RequestParam String column) throws Exception {
    List<Server> servers = new ArrayList<>();

    try (Connection connection = dataSource.getConnection();
            PreparedStatement preparedStatement = connection.prepareStatement("select id, hostname, ip, mac, status, description from servers  where status <> 'out of order' order by " + column)) {
        ResultSet rs = preparedStatement.executeQuery();
        ...

Can SAST Find?

Possible

Can DAST Find?

Possible

SAST Reasoning:

While the SQL query is a prepared statement, the column field can not be specified as a parameter in a parameterized query. A SAST tool should identify that the query string is concatenated with user input on line 73.

DAST Reasoning:

A DAST tool, if it is able to find the /WebGoat/SqlInjectionMitigations/servers endpoint, should be able to exploit the column based SQL injection issue.

Example Attack:

/WebGoat/SqlInjectionMitigations/servers?column=(select+repeat('a',50000000)+from+information_schema.tables)

(A1) Injection > Path traversal > Path traversal while uploading files

Link: http://localhost:8080/WebGoat/start.mvc#lesson/PathTraversal.lesson/1

Source:

webgoat-lessons/path-traversal/src/main/java/org/owasp/webgoat/path_traversal/ProfileUpload.java
webgoat-lessons/path-traversal/src/main/java/org/owasp/webgoat/path_traversal/ProfileUploadBase.java

Lesson:

This lesson's goal is to upload a file to overwrite a system file. It uses a common insecure pattern of using user input to generate a file path and supplying the file contents to create a new file.

webgoat-lessons/path-traversal/src/main/java/org/owasp/webgoat/path_traversal/ProfileUpload.java#L31-L32

public AttackResult uploadFileHandler(@RequestParam("uploadedFile") MultipartFile file, @RequestParam(value = "fullName", required = false) String fullName) {
    return super.execute(file, fullName);
}

webgoat-lessons/path-traversal/src/main/java/org/owasp/webgoat/path_traversal/ProfileUploadBase.java#L41-43

var uploadedFile = new File(uploadDirectory, fullName);
uploadedFile.createNewFile();
FileCopyUtils.copy(file.getBytes(), uploadedFile);
...

Can SAST Find?

Possible

Can DAST Find?

Improbable

SAST Reasoning:

A SAST tool should be able to identify that the new File call's second parameter is tainted from the uploadFileHandler fullName parameter.

DAST Reasoning:

DAST tools have difficulties identifying insecure file upload path traversal attacks. As the attack is a two-step process, the tool most likely will not be able to identify where the uploaded file ultimately resides on the file system. The first step is uploading the file, while the second is for identifying where the uploaded file exists and attempting to access it.

In this lesson there is no way to retrieve a non .jpg suffixed file. The var catPicture = new File(catPicturesDirectory, (id == null ? RandomUtils.nextInt(1, 11) : id) + ".jpg"); limits what can be retrieved. Additionally, the only way to attempt to access uploaded files is through a completely unrelated web page. A DAST tool would not understand the relationship between the upload endpoint and the retrieval endpoint.

While it is possible to attempt null byte attacks if the string contains path-traversal-secret.jpg the FileCopyUtils method uses java.io.File.toPath which disallows null bytes. For example attempting the attack: /WebGoat/PathTraversal/random-picture?id=%2e%2e%2f%2e%2e%2f%2e%2e%2f%2e%2e%2f%2e%2e%2f%2e%2e%2fetc%2fpasswd%00path-traversal-secret.jpg%00 will fail with the error: java.nio.file.InvalidPathException: Nul character not allowed.

Given the above, while it is possible to upload arbitrary files, it is not possible to access them, hence a DAST tool would be unable to identify this as a flaw. A human of course could use various techniques to attempt to overwrite system files, but a DAST tool is not built for such activities.

(A1) Injection > Path traversal > Path traversal while uploading files (lesson 3)

Link: http://localhost:8080/WebGoat/start.mvc#lesson/PathTraversal.lesson/2

Source:

webgoat-lessons/path-traversal/src/main/java/org/owasp/webgoat/path_traversal/ProfileUploadFix.java
webgoat-lessons/path-traversal/src/main/java/org/owasp/webgoat/path_traversal/ProfileUploadBase.java

Lesson:

This lesson's goal is to upload a file to overwrite a system file. It uses a common insecure pattern of using user input to generate a file path and supply the file contents to create a new file. In this lesson filtering is done on user input, replacing ../ with an empty string.

webgoat-lessons/path-traversal/src/main/java/org/owasp/webgoat/path_traversal/ProfileUploadFix.java#L24-L28

    public AttackResult uploadFileHandler(
            @RequestParam("uploadedFileFix") MultipartFile file,
            @RequestParam(value = "fullNameFix", required = false) String fullName) {
        return super.execute(file, fullName != null ? fullName.replace("../", "") : "");
    }

webgoat-lessons/path-traversal/src/main/java/org/owasp/webgoat/path_traversal/ProfileUploadBase.java#L41-43

var uploadedFile = new File(uploadDirectory, fullName);
uploadedFile.createNewFile();
FileCopyUtils.copy(file.getBytes(), uploadedFile);

Can SAST Find?

Can DAST Find?

Improbable

SAST Reasoning:

A SAST tool should be able to identify that the new File call's second parameter is tainted from the uploadFileHandler fullName parameter.

DAST Reasoning:

(A1) Injection > Path traversal > Path traversal while uploading files (lesson 4)

Link: http://localhost:8080/WebGoat/start.mvc#lesson/PathTraversal.lesson/3

Source:

webgoat-lessons/path-traversal/src/main/java/org/owasp/webgoat/path_traversal/ProfileUploadRemoveUserInput.java
webgoat-lessons/path-traversal/src/main/java/org/owasp/webgoat/path_traversal/ProfileUploadBase.java

How to benchmark security tools: a case study using WebGoat

Running a fair benchmark

Choosing the right tool

Choosing applications

Real world flaws

Are results consistent and useful?

WebGoat as a benchmarking target

WebGoat's architecture

Analysis methodology

(A1) Injection

(A1) Injection > SQL Injection (intro) > What is SQL?

(A1) Injection > SQL Injection (intro) > Data Manipulation Language (DML)

(A1) Injection > SQL Injection (intro) > Data Definition Language (DDL)

(A1) Injection > SQL Injection (intro) > Data Control Language (DCL)

(A1) Injection > SQL Injection (intro) > What is SQL injection?

(A1) Injection > SQL Injection (intro) > Try It! String SQL injection

(A1) Injection > SQL Injection (intro) > Try It! Numeric SQL injection

(A1) Injection > SQL Injection (intro) > Compromising confidentiality with String SQL injection

(A1) Injection > SQL Injection (intro) > Compromising Integrity with Query chaining

(A1) Injection > SQL Injection (intro) > Compromising Availability

(A1) Injection > SQL Injection (advanced) > Try It! Pulling data from other tables

(A1) Injection > SQL Injection (advanced) > (no title)

(A1) Injection > SQL Injection (mitigation) > Input validation alone is not enough!! (Lesson 9)

(A1) Injection > SQL Injection (mitigation) > Input validation alone is not enough!! (Lesson 10)

(A1) Injection > SQL Injection (mitigation) > (no title) (Lesson 12)

(A1) Injection > Path traversal > Path traversal while uploading files

(A1) Injection > Path traversal > Path traversal while uploading files (lesson 3)

(A1) Injection > Path traversal > Path traversal while uploading files (lesson 4)

More to explore

Introducing Custom Compliance Frameworks in GitLab

Enhance application security with GitLab + HackerOne

Secure and safe login and commits with GitLab + Yubico

We want to hear from you

Ready to get started?

How to benchmark security tools: a case study using WebGoat

Running a fair benchmark

Choosing the right tool

Choosing applications

Real world flaws

Are results consistent and useful?

WebGoat as a benchmarking target

WebGoat's architecture

Analysis methodology

(A1) Injection

(A1) Injection > SQL Injection (intro) > What is SQL?

(A1) Injection > SQL Injection (intro) > Data Manipulation Language (DML)

(A1) Injection > SQL Injection (intro) > Data Definition Language (DDL)

(A1) Injection > SQL Injection (intro) > Data Control Language (DCL)

(A1) Injection > SQL Injection (intro) > What is SQL injection?

(A1) Injection > SQL Injection (intro) > Try It! String SQL injection

(A1) Injection > SQL Injection (intro) > Try It! Numeric SQL injection

(A1) Injection > SQL Injection (intro) > Compromising confidentiality with String SQL injection

(A1) Injection > SQL Injection (intro) > Compromising Integrity with Query chaining

(A1) Injection > SQL Injection (intro) > Compromising Availability

(A1) Injection > SQL Injection (advanced) > Try It! Pulling data from other tables

(A1) Injection > SQL Injection (advanced) > (no title)

(A1) Injection > SQL Injection (mitigation) > Input validation alone is not enough!! (Lesson 9)

(A1) Injection > SQL Injection (mitigation) > Input validation alone is not enough!! (Lesson 10)

(A1) Injection > SQL Injection (mitigation) > (no title) (Lesson 12)

(A1) Injection > Path traversal > Path traversal while uploading files

(A1) Injection > Path traversal > Path traversal while uploading files (lesson 3)

(A1) Injection > Path traversal > Path traversal while uploading files (lesson 4)

Sign up for GitLab’s newsletter

More to explore

Introducing Custom Compliance Frameworks in GitLab

Enhance application security with GitLab + HackerOne

Secure and safe login and commits with GitLab + Yubico

We want to hear from you

Ready to get started?