How to fuzz a server with American Fuzzy Lop
Senior Principal Engineer, Fastly
American Fuzzy Lop (AFL) is an open source, coverage-assisted fuzz testing tool developed by Michał Zalewski of Google. In a nutshell, it feeds intelligently crafted input to a program that exercises corner cases and finds bugs in a target program.
In this blog post, I'll describe how to use AFL's experimental persistent mode to blow the doors off of a server without having to make major modifications to the server's codebase. I've used this technique at Fastly to expand testing in some of the servers that we rely on and others that we are experimenting with.
Throughout this post, I'll use the open source Knot DNS with a basic configuration as a running example, but the technique is applicable to other servers and long running processes as well.
The problem: File input and startup time
AFL passes input via a file interface, but servers generally read from sockets or other network-ready interfaces. In addition, by default AFL re-runs the target program each time it executes a test. Because servers often take a second or more to start, this can greatly hamper AFL's progress in exploring the program and uncovering bugs.
In the past, testers have had to apply test harnesses that are either non-trivial to develop or harnesses that test only a subset of the server's request handling logic (see the existing AFL harness in Knot DNS for an example). Neither of these solutions is ideal — the first one can be time-consuming (and require heavy grokking of source code) and the latter limits your test coverage.
The solution: Persistent mode
You can read more about AFL's experimental persistent mode here. Via some minor modifications to the source code, it lets the tester control:
When AFL forks the target application
When AFL feeds new input to the application
This mode solves the problem of waiting for a program to start; the tester can now feed fuzzed inputs to the target program without restarting it. The major caveat here is that the tester must be careful to reset the state of the application between each iteration of fuzzing. If the state is not reset carefully, you'll find bugs in your test harness instead of the target.
The insight: Persistent mode + servers
At this point, the good news might be obvious: many servers carefully reset the state of the application each time a request is processed for you, so the "hard part" of using AFL persistent mode is already taken care of. In general, a server thread will look something like this:
while (go):
req = get_request()
process(req)To integrate AFL persistent mode, all you have to do is modify the program to do this:
while (go)
put_request(read(file)) // AFL
req = get_request()
process(req)
notify_fuzzer() // AFLIt turns out (from my experience, anyway) that it takes a lot less time to identify and modify the target code above than to figure out how to write a custom test harness that might involve mocking API functions and/or extracting parsing logic into a standalone program.
Applying the technique to Knot DNS
Finding the processing loop
Knot DNS uses sockets for communication, so I searched the source code for select and found the loop that reads and processes UDP packets. Here is a snippet of the relevant source code (I added **** AFL: .. **** comments to show where changes are necessary to support fuzzing in AFL persistent mode):
// **** AFL: Declare and initialize variables ****
...
/* Loop until all data is read. */
for (;;) {
/* Check handler state. */
if (unlikely(*iostate & ServerReload)) {
*iostate &= ~ServerReload;
udp.thread_id = handler->thread_id[thr_id];
rcu_read_lock();
forget_ifaces(ref, &fds, maxfd);
ref = handler->server->ifaces;
track_ifaces(ref, &fds, &maxfd, &minfd);
rcu_read_unlock();
}
/* Cancellation point. */
if (dt_is_cancelled(thread)) {
break;
}
/* Wait for events. */
fd_set rfds;
FD_COPY(&fds, &rfds);
// **** AFL: Read from input file here ****
int nfds = select(maxfd + 1, &rfds, NULL, NULL, NULL);
if (nfds <= 0) {
if (errno == EINTR) continue;
break;
}
/* Bound sockets will be usually closely coupled. */
for (unsigned fd = minfd; fd <= maxfd; ++fd) {
if (FD_ISSET(fd, &rfds)) {
if ((rcvd = _udp_recv(fd, rq)) > 0) {
_udp_handle(&udp, rq);
/* Flush allocated memory. */
mp_flush(mm.ctx);
_udp_send(rq);
udp_pps_sample(rcvd, thr_id);
}
}
}
// **** AFL: Notify fuzzer that processing complete here ****
}Note: I targeted select for Knot DNS, but if your target server uses a different API for handling network traffic you should be able to search for its analogous recv or read functions to accomplish the same end.
The shims
I call each of the modifications that I made to support persistent mode a shim. The first shim is pretty boring; it just declares and initializes variables needed for the other shims:
#ifdef KNOT_AFL_PERSISTENT_SHIM /* For AFL persistent mode fuzzing shim */
/* Initialize variables for fuzzing */
size_t insize;
struct sockaddr_in servaddr;
int udp_socket;
char *env_dest_ip = getenv("KNOT_AFL_DEST_IP");
char *env_dest_port = getenv("KNOT_AFL_DEST_PORT");
int dest_port = env_dest_port ? strtol(env_dest_port, NULL, 10) : 9090;
char *dest_ip = env_dest_ip ? env_dest_ip : "127.0.0.1";
bzero(&servaddr,sizeof(servaddr));
servaddr.sin_family = AF_INET;
servaddr.sin_addr.s_addr = inet_addr(dest_ip);
servaddr.sin_port = htons(dest_port);
char buf[5120];
#endif // #ifdef KNOT_AFL_PERSISTENT_SHIM Note: the code above reads the destination IP address and port from environment variables with some defaults. This is important if you plan to scale up an AFL run — you'll probably want each instance of the server to listen on a different port in that case.
The second shim reads fuzzed input from a file — stdin in this case — and feeds it to one of the sockets that Knot DNS is listening on.
#ifdef KNOT_AFL_PERSISTENT_SHIM /* For AFL persistent mode fuzzing shim */
/* Read fuzzed packet from stdin and send to socket */
if (getenv("KNOT_AFL_STDIN") || getenv("KNOT_AFL_CMIN") ||
getenv("AFL_PERSISTENT")) {
memset(buf, 0, 5120);
insize = read(0, buf, 5120);
udp_socket = ((iface_t*)HEAD(handler->server->ifaces->l))->fd[IO_UDP];
sendto(udp_socket, buf, insize,0, (struct sockaddr *)&servaddr,sizeof(servaddr));
}
#endif // #ifdef KNOT_AFL_PERSISTENT_SHIMI usually try to make this section of the code stateless so that I don't have to worry about cleaning anything up. If you do make any allocations here, it is important to make sure they are cleaned up in the post-processing shim (see below), taking care to consider any edge cases that might occur when a message is processed.
Note that this code does require a little thinking. In this case, I use a Knot-specific data structure to grab the file descriptor for a socket that the server is going to read from. In other cases I've had to use FD_ISSET and a one-liner loop to find a socket that is passed to select in the readfds set. It's not usually difficult regardless — if you are coding near select the set of file descriptors it reads from can't be too far away.
Also note the use of getenv here. This shim only executes if KNOT_AFL_STDIN is set (used for smoke testing), KNOT_AFL_CMIN is set (I use this for afl-cmin runs), or AFL_PERSISTENT is set (this is set in AFL persistent mode).
The third shim is pretty basic. It simply signals AFL that a fuzz iteration is complete (via a SIGSTOP), exits so the next seed packet can be processed if this is an afl-cmin run, or does nothing:
#ifdef KNOT_AFL_PERSISTENT_SHIM /* For AFL persistent mode fuzzing shim */ /* Signal AFL to fuzz input and continue execution */
if (getenv("AFL_PERSISTENT")) {
raise(SIGSTOP);
} else if (getenv("KNOT_AFL_CMIN")) {
exit(0);
}#endif // #ifdef KNOT_AFL_PERSISTENT_SHIMI used preprocessor macros to conditionally include all of the shims above. This way, while the shims still muck up the codebase a little, at least they won't interfere with any other builds.
Fuzzing
You can learn more about fuzzing with AFL and using persistent mode in the documentation, but I've included some examples of getting AFL running with this harness below.
Configure and compile the app
$ CC=~/afl-1.83b/afl-clang-fast CFLAGS='-DKNOT_AFL_PERSISTENT_SHIM' ./configure --disable-shared
$ makeNote: you'll have to compile afl-clang-fast from the llvm_mode directory to support persistent mode; see the README for more info.
Minimize your test cases
KNOTD_AFL_CMIN=1 ~/afl-1.83b/afl-cmin -i ~/knot-seeds -o ~/knot-seeds-cmin -- ~/knot-dns/src/knotd -c my_config.configStart fuzzing in persistent mode
AFL_PERSISTENT=1 ~/afl-1.83b/afl-fuzz -i ~/knot-seeds-cmin -o ~/my_output_dir ~/knot-dns/src/knotd -c my_config.configAFL persistent mode is kind of awesome, but you don't have to take my word for it
If you've made it this far, hopefully you now have an idea of how to apply AFL persistent mode to a server to significantly increase your fuzzing coverage. While I used Knot DNS as an example, this technique should be applicable to most servers or daemons that use a read-and-process loop pattern. With a little study, even servers that carry state beyond the protocol level should be addressable with persistent mode.
Footnote: So where are the bugs?
The Knot team was already using AFL to fuzz a subset of their server logic (and I've sent a persistent mode patch upstream to them), but it’s worth noting that I was able to find vulnerabilities in another popular DNS server in a matter of hours on a low-end PC using this technique (we're coordinating a fix with the upstream vendor so I'm holding off on discussing that for the time being). Update on August 4, 2015: Scroll down for info on the fix.
Regardless, to give a completely unscientific example of the testing gains that AFL persistent mode can bring to a project, here is the lcov/genhtml coverage summary resulting from fuzzing Knot DNS with its existing test harness for five minutes under AFL in a VM on a laptop:
Overall coverage rate:
lines......: 24.5% (711 of 2903 lines)
functions..: 28.1% (88 of 313 functions)For comparison, here is the coverage summary from running with the new harness for five minutes using a simplified config file:
Overall coverage rate:
lines......: 23.9% (6638 of 27796 lines)
functions..: 32.9% (704 of 2139 functions) While some of the coverage reported in the persistent mode run is initialization logic, the line and function coverage reached by the tests increased significantly with the new harness. (Note: if you are confused by the percentages, consider that they are scaled by the size of the program.) In addition, the lines that are covered by AFL are getting executed considerably more using the new harness. The AFL status screen shows the following when the five minute experiment using the existing harness exits:
total execs : 518k
exec speed : 1610/sec And the following when the new harness exits:
total execs : 1.34M
exec speed : 3382/secSo, while you're not going to find 0day in this blog post, hopefully the potential gains in coverage, executions, and the relative simplicity of this technique will motivate you to give it a try. Thanks for reading!
Update on August 4, 2015: After coordinating with the vendor (ISC), this vulnerability has since been publicly disclosed. I used the technique described in this blog post to find CVE-2015-5477, a critical vulnerability in Bind. You can read more about the vulnerability here and here.
