About Arctic Engine
Arctic Engine is an open-source, free game engine released under the MIT license. Arctic Engine is implemented in C++ and focuses on simplicity. Being a C++ programmer and making games should not be joyless, disillusioning, and discouraging. In the '80s and '90s, a programmer could make games alone, and it was fun. Arctic Engine aims at making game development in C++ fun again.
Testing can be fun
Testing the game engine is very important since games are usually no more robust and performant than the underlying middleware or game engine. Writing tests by hand is time-consuming and disillusioning, and it may drain the fun from the development process. So, to my shame, I avoided writing tests in every way I could. For instance, I used static analyzers to detect bugs. The problem with static analyzers was the lack of motivation to fix potential issues. You may be unsure whether a bug is really there, and it can sometimes be hard to find a way to trigger it.
The other possibility was fuzz testing. I heard about fuzzing but didn't try it earlier because I thought it was hard to integrate with the project. I could not be more wrong. It's amazing how little effort it takes to get fuzz testing up and running with GitLab.
Fuzz testing and what it exposed
Thanks to Sam Kerr for proving me wrong about fuzzing by actually fuzzing the sound loader code. Arctic Engine allows loading a sound from a WAV file in memory. To fuzz the loader's code, you create a small CPP file with a single function like this:
extern "C" int LLVMFuzzerTestOneInput(const uint8_t * data, size_t size) {
std::shared_ptr<arctic::SoundInstance> result = arctic::LoadWav(data, size);
return 0;
}
Then you add -fsanitize=fuzzer flag to the CMakeLists.txt file and a few
lines to the .gitlab-ci.yml file, and the fuzzing begins! You may want to
drop in a few WAV files to the corpus folder to help the fuzzer and speed up
the process, but that's optional. Ok, it was a little harder than that with the
Arctic Engine because it would output a message and quit upon processing
unsupported file formats. Still, handling file loading errors this way was a
bad idea, and I finally had a reason to fix it.
The fuzzer started crashing Arctic Engine: first, it triggered a signed integer overflow, a division by zero, and a buffer overrun. And then, the wave loader got out-of-memory while trying to resample a tiny WAV file with a sampling rate of 1 sample per second to 44100 samples per second. Wow.
What I liked about fuzzing is that fuzzer actually crashes your program and provides you the input so you can reproduce the crash. And once you've set up the test harness, the entire testing process is fully automated, saving you time and effort. It's like having a personal QA team, you commit your code, and in a few minutes, you already have it tests-covered.
Then I fuzzed the CSV and the TGA file parsers and expected to find some bugs in the CSV and none in the TGA. What can I say? You may not find bugs where you expect them to be and find bugs where you thought there were none. The TGA loader crashed immediately with a buffer overrun. It did not account for files containing only a valid header but no actual image data after it.
Plans
I will add a simple HTTP web server and some multiplayer network interaction code to the Arctic Engine. I was putting it off for quite a while now because I thought testing would be a pain. Now that I know how easy it is to apply GitLab's fuzz testing to any data processing code, I'm very optimistic and somewhat challenged. Like "Can I make it withstand the fuzzer from the first try?". It makes writing code fun for me once again.
Further reading
About the guest author
Huldra is a senior videogame programmer by day maintainer of the Arctic Engine by night. She started it because she wanted a game engine that kept simple things simple and made complex things possible.