Fuzz testing is a novel way to discover security vulnerabilities or bugs in software applications. Unlike traditional software testing methodologies – SAST, DAST or IAST – fuzz testing essentially “pings” code with random (or semi-random) inputs in an effort to crash it and thus identify “faults” that would otherwise not be apparent. Those code faults (or problems with business logic) represent areas that are potentially at high risk for security threats. The name “fuzz” is a reference to the random nature of the process. Fuzz testing’s supporters praise it for being fully-automated and able to find obscure weaknesses, while it’s detractors complain it can be difficult to set up and prone to deliver unreliable results.
Fuzz testing stands out in another way as well: there’s actually a story about how the concept was discovered. In 1988, University of Wisconsin – Madison Professor Barton Miller was trying to access code remotely using a dial-up system, but feedback from a thunderstorm kept causing the program to crash. The idea that external “noise” couldn’t be tolerated by code became the inspiration for Miller and his student’s work. They discovered that Unix, Mac, and Windows programs would routinely crash when pinged by random unexpected inputs. Miller is one of the authors of Fuzzing for Software Security Testing and Quality Assurance.
There are two main types of fuzz testing: coverage-guided and behavioral.
Coverage-guided fuzz testing focuses on the source code while the app is running, probing it with random challenges in an effort to uncover bugs. New tests are constantly being generated and the goal is to get the app to crash. A crash means a potential problem, and data from the coverage-guided fuzz testing process will allow a tester to reproduce the crash which is helpful when trying to identify at-risk code.
Behavioral fuzz testing works differently. Using specs to show how an application should work it uses random inputs to judge how the app really works; the difference between the expected and the reality is generally where bugs or other potential security risks can be found.
Because of the random nature of fuzz testing, experts say it’s the methodology most likely to find bugs missed by other tests. It’s also seen as an incredibly low-effort testing methodology, or what some like to call “set it and forget it.” Once the test harness is created fuzz testing is fully automated and will run indefinitely. It can be scaled easily by spinning up more machines and is a good choice for regression testing.
Fuzz testing is also ideal to work alongside a manual testing team as both sets of inputs will educate the other.
Two main challenges face practitioners looking to implement fuzz testing: setup and data analysis. Fuzz testing isn’t necessarily easy to set up; it requires complex testing “harnesses” that can be even more tricky to create if the fuzz testing isn’t actually located within an existing toolchain. Plus, fuzz testing can generate a lot of data, including potentially false positives. So it’s critical to ensure that a testing team is prepared to deal with the onslaught of information.
Also, while less easy to document, negative attitudes toward the “vague” nature of fuzz testing do persist in the QA community.