In this post we will focus on why continuous fuzzing is needed and what are the challenges in implementing continuous fuzzing. Previous posts/papers regarding why fuzzing in general is important as well as why it’s a good idea even to integrate it as part of the Go toolchain can be found here (written by Dmitry Vyukov and Romain Baugue – highly recommended, the link talks about go but concepts can be applied to other languages).
Fuzzing Quick Recap?
Essentially fuzzing consist of two types of jobs:
- Fuzzing – A job that can run infinitely. This job automatically generates interesting test-cases that cover more paths, as well as monitors for crashes and other memory related problems.
- Regression – Run the fuzzer through a set of specific test-cases. Usually these were generated by the previously mentioned long running jobs. Regression jobs are generally very short.
Continuous Fuzzing Challenges
There are a few challenges/questions that arise from how to integrate fuzzing to the current CI. We will walk through some of them as there are a lot of other open questions that really depends on the development workflow and the specific project.
Challenge 1 – Long Running Jobs: Fuzzing is a long-running (infinite) job unlike a CI that we try to keep as short as possible to provide fast feedback for commits/MRs.
Solution 1 – Async Jobs: This where we need to spawn a different server or use a platform like GitLab to run the fuzzers asynchronously. The platform will notify the administrators, developers, or the relevant security people of any new vulnerabilities that the fuzzers find. In GitLab, these will be reported on our Security Dashboard. This could take days or months of running continuously.
Challenge 2 – Many Targets * Many Versions = Lots of time and money: Which versions should should you fuzz? We need to decide wisely which versions to fuzz, as blindly fuzzing all possible versions in a project infinitely for many targets will cost a lot of money and compute resources.
Solution 2 – Master + Stable: One approach that we saw popular with users is fuzzing the development branch (master) and release branch. The development branch is fuzzed continuously and the fuzzer is updated every time new code is pushed to master. The updated fuzzers check the additional code but keeps the corpus from previous runs. This way the fuzzers can essentially always continue from where they stopped and only work on the additional code. GitLab helps with managing the corpus and keeping it in minimised state.
Challenge 3 – Learning from old mistakes: Once we setup continues fuzzing we aggregate very valuable test-cases and crashes that we fix along the way. We would love to use all those precious test-cases to check every MR before it get’s merged.
Solution 3 – Regression Fuzz Tests: For every MR just like unit-test we run the fuzzers through all the generated test-cases and the fixed crashes which is usually a very quick process which fits a classic CI. GitLab helps running the fuzzers with the aggregated corpus from previous job, fail the CI and alert the developer immediately via the security dashboard. Short regresion can also be combined with short fuzz tests runs that run inline with the CI to help find also new bugs in MRs.
Summary
This was a quick walkthrough of the some of the challenges of integrating continuous fuzzing to projects from our experience.
Check out our full documentation and the example repositories and try adding fuzz testing to your own repos!