Fuzz testing, or fuzzing, is the act of inputting unexpected, malformed, and/or random data to measure response or stability of an application or service. This is accomplished by monitoring for unexpected behavior or service / application crashes. Fuzz testing will find issues that traditional testing and QA methods typically do not.
The goal of fuzz testing is to discover software defects that need to be addressed as these defects may lead to (exploitable) vulnerabilities that other QA methods and security scanning miss. Fuzz testing can find not only security issues, but also flaws in the business logic of an application or service.
GitLab offers two types of fuzz testing:
Coverage-guided fuzz testing is all about using contextual information from source code to better inform fuzz tests as well as to help correlate the results of a fuzz testing crash directly to the region of code that was vulnerable. This dramatically improves the cycle time to go from an initial fuzz test to a crash to an update to vulnerable areas.
A large benefit of coverage-guided fuzz testing is that it can be done without requiring all development of an app to be completed or for a Review App to be created for live testing. Instead, coverage-guided fuzz testing can be done iteratively on small parts of the app. This means that you can integrate fuzz testing earlier into your SDLC and shift security further left. Getting these results sooner means that developers can act on them sooner, reducing cycle times.
Traditional fuzz testing is a technique that uses a definition of the inputs a target application is expecting to better understand the Implementation Under Test (IUT). This allows the fuzzer to smartly make mutations that are very close to valid, in contrast to fuzzers that don't understand the expected inputs of the IUT are.
Additionally, traditional fuzz testing is able to observe how the behavior of the IUT changes to make different decisions for subsequent fuzz tests. For example, if a traditional fuzz test notices that a specific type of input causes HTTP 500 errors, it could make similar sorts of mutations to find errors in the same parts of the app. This approach provides better quality of results by inducing more faults and helping to pinpoint where they are in the app.
Fuzz testing can be performed using one of the two fuzz algorithm strategies:
GitLab will provide more valuable fuzz testing results by combining both traditional and coverage-guided fuzz testing techniques. By using both approaches together, we can find more faults and vulnerabilities than one approach would find on its own.
Traditional fuzz testing tests what an app should be doing, based on its published interfaces. Because we also have the source code of the app available at the time of test, we can use coverage-guided fuzz testing techniques to identify parts of the app that aren't mentioned in the publicly facing interfaces. We can also use the source code of the app to provide context and better understand why the app is crashing. This allows us to intelligently generate new test cases to reach other parts of the app's code.
A final, and very important, benefit of combining the techniques is that because we have context around where in the code a fault has occurred, we can point users to specific parts of their app where the fault occurred so it can be fixed. Generally, the process of determining where in source code a fault occurred is difficult, so allowing users to immediately see this location is incredibly valuable and will directly help them to mitigate identified issues.
We have created a playlist of videos that discuss the different types of fuzz testing, techniques, and how to use fuzz testing called Fuzzing 101 that has lots of information you may find useful.
In a survey GitLab conducted in May 2020, 81% of our respondents said that they think fuzz testing is important. However, over 60% said that the most difficult part of fuzz testing is setting it up and integrating it into their CI systems. Because of this difficulty, only 36% of respondents were actually using fuzz testing.
Despite many people thinking fuzz testing is important, the large number of users not using it shows that there is an opportunity to give new value with GitLab if we can help users overcome those initial pain points related to adopting fuzz testing.
There are two primary target audiences for fuzz testing:
There is an additional role which may be included in the target audience. Product verification (PV) engineers may leverage fuzz testing as part of their unit or regression tests. If the software development team does not directly handle their unit test creation, a PV engineer may be partnered with a software development team to create and maintain unit tests on behalf of the software development team.
Sasha is the primary persona we will focus on building fuzz testing for initially. Ensuring they are able to successfully set up and consume the results of a fuzz test is critical to drive adoption of fuzz testing.
We prefer to get 80% of fuzz testing quality while only requiring users like Sasha to do 20% of the work. Even though better results are possible with more work by the end-user, we do not want to go in this direction initially since it means users will get frustrated and not use fuzz testing at all.
In the future, we will focus on building out deeper fuzz testing workflows to support Sam and the security research they are doing. We will also deepen the fuzz testing experience so that users like Sasha can easily learn more about fuzz testing and become a more advanced user without requring a large amount of time to be invested. At that point, we will be focusing on getting the 20% of better quality results that require the 80% of work by end users.
Approaching and prioritizing personas in this way will help us to drive adoption of Fuzz Testing and address the pain points identified in our research.
GitLab believes there are different types of markets and users that will find value in fuzz testing. To this end, we are prioritizing what use cases and where we want to focus first. Read more about them in full on our market segments page.
As we mature fuzz testing, we will work closely with existing GitLab users to add fuzz testing to their apps. If users already have Source Code Management and CI/CD set up for their projects, this will make it easy for them to add fuzz testing to what they already have. It will be more work for a brand new user to GitLab to add SCM, CI/CD, and our other capabilities in addition to also adding fuzz testing. Getting up and running with the other capabilities can be done quickly with our quick start guides, examples, and creating projects with AutoDevOps, but those will be extra steps to complete before adding fuzz testing.
Focusing on existing users will help us move quicker in getting users onboarded with fuzz testing and will also help increase our Stages per User metric.
In keeping with GitLab's seed then nurture, we are focused on adding support for different technologies, languages, and use cases initially to ensure fuzz testing can be broadly adopted by all our users. Over time, we will build deeper experiences and more capabilities where they will be most impactful.
GitLab's aspirations with fuzz testing is that it becomes a security technique that all of our users take advantage of to find and fix issues in their apps before attackers can exploit them.
Since fuzz testing is traditionally difficult to use, our emphasis is on making fuzz testing an accessible technology that our users can take advantage of without becoming technical experts in the space nor requiring them to form a dedicated fuzz testing team.
GitLab will provide its users with application robustness and security visibility testing solutions which validate the entire application stack. This will be provided by verifying the user’s application or service both while at rest as well as while running. This will include historical trending and recommendations for next steps to provide peace of mind to our users.
Furthermore, GitLab will provide real-time remediation for user solutions running in production. This will be possible by analyzing the user issues found in GitLab Secure and applying “virtual patches” to the user’s production application or service leveraging GitLab Defend. This will allow your organization to be secure and continue functioning until the underlying vulnerability can be remediated in your code.
GitLab ultimately envisions our fuzz testing solutions becoming a critical part of every organization's DevSecOps workflow, whether that is for web APIs, traditional desktop software, or for hardware devices. Our long-term vision is to provide shift fuzz testing solutions left for these use cases and that fuzz testing becomes a common part of every developer's workflow. We will start with web API and web appliation use cases, which are already well supported in GitLab, as part of an iterative approach to delivering fuzz testing.
Our near-term goals are to drive additional adoption of fuzz testing and deliver the capabilities needed to drive that adoption.
Since we are moving forward quickly and have limited resources, it is important we focus our efforts where they will be most impactful. To this end, our high-level near-term priorities are:
We have several longer-term, more strategic objectives we will be focusing on as well. Those are discussed in more detail in our longer-term strategy page.
Our initial focus is on web applications and REST APIs, so we are not focusing on fuzz testing local desktop or mobile applications at this time.
Applications that require special hardware, such as Wireless, Bluetooth, and Automotive-based fuzz testing, are not where we are focused now. This is because requiring proprietary hardware is not where GitLab is strong today and makes it more difficult for us to quickly iterate.
The fuzz testing competitive landscape is composed of both commercial and open source testing solutions. The following outlines the competitive landscape for fuzz testing solutions:
GitLab already uses OWASP ZAP today to power the DAST capability within the GitLab Secure offerings. ZAP is a web application security scanner, leveraging a proxy to perform testing, and does not perform protocol-level fuzz testing. All fuzz testing by ZAP is performed within the context of the web application being tested.
GitLab announced the acquisition of Fuzzit and Peach Tech on June 11, 2020. This immediately enhanced and accelerated our plans for fuzz testing. Watch the video below to get all the details!
Analysts consider fuzz testing to be part of Application Security Testing and is generally discussed as part of those reports, rather than as a standalone capability. There are challenges to make it part of the DevSecOps paradigm, and they are interested to hear more about that.
Gartner wrote "Outsourcing or leveraging managed services to perform fuzzing is the recommendation in the absence of internal subject matter expertise and staffing." as part of their "Structuring Application Security Practices and Tools to Support DevOps and DevSecOps" research. This recommendation is a result of the high-level of manual configuration many fuzzers need. This is a pain point GitLab should be mindful of as we work on bringing our own fuzz testing solutions out so that users can be successful without needing to be a fuzz testing expert nor need to do a large amount of configuration.
Gartner also published their How to Deploy Application and Perform Application Security Testing where they say "You should reserve fuzzing for nonweb applications." We disagree with this conclusion and think that it comes primarily from the difficulty normally associated with doing fuzz testing. This underscores the importance of our emphasis of making fuzz testing straightforward and easy to use.
Additionaly, as APIs become more and more of a staple of modern application development, they should be prioritized for fuzz testing. As one of the primary interfaces exposed to end-users, fuzz testing can reveal critical bugs and security issues before they are exposed to the world, despite them not being traditional web applications.
By combining our fuzz testing solution with our DAST, IAST, and SAST solutions into a GitLab Secure suite, GitLab can be placed into the Gartner Magic Quadrant for Application Security Testing. This will give GitLab additional exposure as well as show security thought leadership.
Other analyst firms include 451 Research and IDC as they have focused security practices in which GitLab Secure can be highlighted and show leadership.
We will continue to update this section with customer success and sales issues as we begin getting feedback.
We will continue to update this section with specific GitLab issues as we begin getting feedback.
We have begun working with internal teams to add fuzz testing in various places, such as GitLab Runner. Working with internal GitLab teams is giving us good feedback is also finding bugs to fix, which means fuzz testing is providing value.
