Hack It Fast and Fix It Forever with Fuzzing

Key Points

• Define Precise Entry Points for Maximum Coverage
• Leverage AFL to Uncover Hidden Bugs
• Turn Fuzzer Output into Long-Term Bug Fixes

What Is Fuzzing?

Fuzzing is a powerful cybersecurity and testing technique that provides randomized inputs to a program to identify vulnerabilities, robustness issues, and edge cases. By introducing unexpected or malformed data, fuzzing can expose critical bugs, including security flaws and crashes.

This guide focuses on American Fuzzy Lop (AFL), a widely used and straightforward fuzzer. AFL is effective for uncovering hidden software issues and improving program resilience.

Create an Entry Point

AFL works by running a unix instrumented binary that processes inputs from stdin or from a file. In order to fuzz a library, a main function must be written first. How this main function is written determines how much of the library’s functionality the fuzzer can explore.

Tips for Crafting Entry Points:

• Maximize Coverage: Ensure the entry point exercises as much of the program’s logic as possible.
• Minimize Restrictions: Allow AFL to explore a wide input space. Avoid limiting the range of inputs at this stage.

Run AFL

To generate an instrumented binary for fuzzing, compile it using AFL’s custom compilers: afl-gcc or afl-clang. These compilers insert additional instructions that monitor code coverage. To detect memory-related errors and undefined behaviors, one should also enable the corresponding sanitization flags:

afl-clang main.c -o fuzz_target -fsanitize=address,undefined

Initial Setup

Provide AFL with valid initial inputs to kickstart its exploration process. For example:

echo -n "0" > init_inputs/test0

The more diverse and meaningful the inputs in init_inputs, the better AFL will perform in discovering complex bugs.

Start Fuzzing:

Launch AFL-Fuzz with the following command:

AFL_SKIP_CPUFREQ=1 AFL_I_DONT_CARE_ABOUT_MISSING_CRASHES=1

afl-fuzz -i init_inputs -o afl_inputs ./fuzz_target

Fix Bugs Found by AFL

After fuzzing, AFL organizes results into two key directories within afl_inputs:

• crashes/: Inputs that caused program crashes (one per unique crash).
• queue/: Non-crashing inputs that expanded code coverage.

Analyzing Crashes:

To reveal the sanitizer error associated with a crash, re-run the program with the crashing input:

./fuzz_target afl_inputs/crashes/input_name

Use a debugger for Root Cause Analysis

Leverage gdb to step through the program and pinpoint the underlying root cause of the bug, instead of just the immediate symptom.

gdb ./fuzz_target

(gdb) run afl_inputs/crashes/input_name

Once the root cause is identified, implement a robust fix or enhance error handling to prevent similar issues from occurring in the future.

Reveal Bugs Beyond Sanitizer Detection With TIS-Analyzer

Modern runtime sanitizers can miss elusive bugs, especially those linked to subtle undefined behaviors. To address these bugs, consider integrating TIS-Analyzer, an advanced analysis tool that applies formal methods to detect undefined behavior in both C and C++ code. By simulating code execution directly from the source, TIS-Analyzer thoroughly examines the code’s semantics, ensuring its validity across all standard-compliant compilers. This proactive approach helps uncover hidden issues that traditional testing and fuzzing might overlook, ultimately improving code reliability, portability, and maintainability. One slogan is that this makes the code “safe for the next twenty years”, identifying misuse of the language that current compilers happen to translate to code that works or that seems to work. But plenty of latent bugs that can have consequences now with a sufficiently sophisticated attacker come up too.

Example of a Commonly Missed Bug:

The condition if (buffer + len > endptr) is never definedly true: it triggers undefined behavior in the circumstances where the developer would expect it to be true. A number of dangers result from this undefinedness.

First, an extremely smart and aggressive optimizing compiler could optimize this condition to always false (remove the check entirely). In 2024, compilers are not aggressive enough to do this, even if you try to trip them up with a buffer statically allocated as an array of a known size. This lack of enthusiasm from compilers may be backlash after a certain CERT advisory against GCC and subsequent LWN article: https://lwn.net/Articles/278137/

Second, and more immediately threatening, the assembly instructions that the compiler generates for if (buffer + len > endptr) use assembly addition without any overflow check for buffer + len and unsigned comparison for >. As a result, if buffer happens to be within 255 bytes of the highest address (by “highest address”, we mean for instance 0xffffffff on a 32-bit platform) and endptr between buffer and 0xffffffff, malicious data can declare 256 as the size of the next sub-object, causing the assembly instructions to decide that buffer + len is very small and definitely less than endptr, that the test is false, and that 256 bytes of data should be processed starting at buffer. This will not end well. Addresses between endptr and 0xffffffff will be accessed, wrongly (who knows what secrets contained there will be leaked?). Eventually, the next address to access will be computed as 0, causing a crash—a denial of service in itself, but perhaps something worse will already have happened before.

When executing compiled code containing the above bug, most often, nothing noticeable happens, because buffer is not within 255 bytes of the end of the address space. Even if the code is instrumented with ASan(*) or UBSan(**), there is nothing to detect, so they do not report anything.

(*) in one of the rare problematic memory layouts, ASan detects that eventually  “processing len bytes at address buffer” does something wrong, but in a non-problematic memory layout there is nothing to detect

(**) in fact, Will Dietz and John Regehr implemented a quick instrumentation to detect that pointer arithmetic leads to the kind of overflow described above in https://reviews.llvm.org/D33305 . This instrumentation ended up in UBSan. Again, in a non-problematic memory layout, there is nothing to detect, the instrumentation only detects the bug if execution happens in one of the rare but not nonexistent problematic layouts.

A corrected condition that ensures defined behavior and generates proper assembly code after compilation is: if(len > (size_t)(endptr - buffer))

Such hidden, exploitable bugs can pass fuzzer tests that rely solely on sanitizers. However, TIS-Analyzer can expose them early using the AFL-generated inputs.

Leveraging TIS-Analyzer to Ensure Robustness and Uncover Root Causes

TIS-Analyzer provides a powerful tool for developers aiming to validate their code against hidden vulnerabilities and undefined behaviors (UBs). Unlike traditional tools, it can identify subtle issues that arise in varying conditions, such as differences in memory layouts or compiler optimization. For instance, a program might crash under specific configurations when executed directly, yet exhibit no issues when run under debugging environments like GDB. This discrepancy can obscure the root cause of a bug, leaving developers puzzled and prolonging the debugging process. TIS-Analyzer eliminates this uncertainty. When an input is successfully analyzed without crashes, developers gain confidence that the program operates safely and predictably with any memory layout and across all standard-compliant compilers and optimization levels (e.g., gcc -O0, clang -O3).

This capability is particularly critical when handling undefined behaviors that are notoriously elusive and multifaceted. Traditional tools, such as sanitizers, often specialize in detecting specific categories of UBs, which can introduce incompatibilities. For example, AddressSanitizer (ASan) may flag an out-of-bounds memory access, while MemorySanitizer (MSan) detects uninitialized memory reads. However, the interplay between these issues is not always straightforward. Does the uninitialized memory read under MSan contribute to the out-of-bounds access under ASan? Or does the root cause lie in a value propagated through calculations, triggering both? Identifying the causal relationship between these events can be very difficult using conventional tools.

Furthermore, tools like GDB have their limitations. Debuggers cannot easily trace back the sequence of events leading to a failure, forcing developers to painstakingly analyze the last point where a variable’s value was modified. The inability to "rewind" execution often results in a trial-and-error process, which is time-intensive and inefficient. By contrast, TIS-Analyzer's GUI allows developers to simulate execution paths comprehensively, pinpointing the exact origin of errors with precision. This not only saves time but also provides actionable insights into the root causes of bugs, ensuring they are resolved effectively.

Integrating TIS-Analyzer into the development workflow ensures that high-coverage inputs from afl_inputs/queue are validated for the absence of undefined behavior. It also helps uncover the root causes of bugs triggered by inputs from afl_inputs/crashes and verifies the efficacy of applied fixes through reanalysis. This holistic approach streamlines the debugging process, enhances code reliability, and fortifies the software against potential vulnerabilities.

Conclusion

By incorporating fuzzing into your development lifecycle, you can expose and address hidden vulnerabilities, ensuring higher-quality, more secure software. The combination of AFL to find relevant code paths, and TIS-Analyzer to find all undefined behaviors and their root cause on those code paths, is a straightforward way to hack fast and fix permanently the bugs of your software system.

For an in-depth exploration of how fuzzing works and its underlying principles, check out this comprehensive guide:

For a deeper understanding and code samples explaining how formal methods improve cybersecurity and critical software testing: