1# Fuzzing with AFL++ 2 3The following describes how to fuzz with a target if source code is available. 4If you have a binary-only target, go to 5[fuzzing_binary-only_targets.md](fuzzing_binary-only_targets.md). 6 7Fuzzing source code is a three-step process: 8 91. Compile the target with a special compiler that prepares the target to be 10 fuzzed efficiently. This step is called "instrumenting a target". 112. Prepare the fuzzing by selecting and optimizing the input corpus for the 12 target. 133. Perform the fuzzing of the target by randomly mutating input and assessing if 14 that input was processed on a new path in the target binary. 15 16## 0. Common sense risks 17 18Please keep in mind that, similarly to many other computationally-intensive 19tasks, fuzzing may put a strain on your hardware and on the OS. In particular: 20 21- Your CPU will run hot and will need adequate cooling. In most cases, if 22 cooling is insufficient or stops working properly, CPU speeds will be 23 automatically throttled. That said, especially when fuzzing on less suitable 24 hardware (laptops, smartphones, etc.), it's not entirely impossible for 25 something to blow up. 26 27- Targeted programs may end up erratically grabbing gigabytes of memory or 28 filling up disk space with junk files. AFL++ tries to enforce basic memory 29 limits, but can't prevent each and every possible mishap. The bottom line is 30 that you shouldn't be fuzzing on systems where the prospect of data loss is 31 not an acceptable risk. 32 33- Fuzzing involves billions of reads and writes to the filesystem. On modern 34 systems, this will be usually heavily cached, resulting in fairly modest 35 "physical" I/O - but there are many factors that may alter this equation. It 36 is your responsibility to monitor for potential trouble; with very heavy I/O, 37 the lifespan of many HDDs and SSDs may be reduced. 38 39 A good way to monitor disk I/O on Linux is the `iostat` command: 40 41 ```shell 42 $ iostat -d 3 -x -k [...optional disk ID...] 43 ``` 44 45 Using the `AFL_TMPDIR` environment variable and a RAM-disk, you can have the 46 heavy writing done in RAM to prevent the aforementioned wear and tear. For 47 example, the following line will run a Docker container with all this preset: 48 49 ```shell 50 # docker run -ti --mount type=tmpfs,destination=/ramdisk -e AFL_TMPDIR=/ramdisk aflplusplus/aflplusplus 51 ``` 52 53## 1. Instrumenting the target 54 55### a) Selecting the best AFL++ compiler for instrumenting the target 56 57AFL++ comes with a central compiler `afl-cc` that incorporates various different 58kinds of compiler targets and instrumentation options. The following 59evaluation flow will help you to select the best possible. 60 61It is highly recommended to have the newest llvm version possible installed, 62anything below 9 is not recommended. 63 64``` 65+--------------------------------+ 66| clang/clang++ 11+ is available | --> use LTO mode (afl-clang-lto/afl-clang-lto++) 67+--------------------------------+ see [instrumentation/README.lto.md](instrumentation/README.lto.md) 68 | 69 | if not, or if the target fails with LTO afl-clang-lto/++ 70 | 71 v 72+---------------------------------+ 73| clang/clang++ 3.8+ is available | --> use LLVM mode (afl-clang-fast/afl-clang-fast++) 74+---------------------------------+ see [instrumentation/README.llvm.md](instrumentation/README.llvm.md) 75 | 76 | if not, or if the target fails with LLVM afl-clang-fast/++ 77 | 78 v 79 +--------------------------------+ 80 | gcc 5+ is available | -> use GCC_PLUGIN mode (afl-gcc-fast/afl-g++-fast) 81 +--------------------------------+ see [instrumentation/README.gcc_plugin.md](instrumentation/README.gcc_plugin.md) and 82 [instrumentation/README.instrument_list.md](instrumentation/README.instrument_list.md) 83 | 84 | if not, or if you do not have a gcc with plugin support 85 | 86 v 87 use GCC mode (afl-gcc/afl-g++) (or afl-clang/afl-clang++ for clang) 88``` 89 90Clickable README links for the chosen compiler: 91 92* [LTO mode - afl-clang-lto](../instrumentation/README.lto.md) 93* [LLVM mode - afl-clang-fast](../instrumentation/README.llvm.md) 94* [GCC_PLUGIN mode - afl-gcc-fast](../instrumentation/README.gcc_plugin.md) 95* GCC/CLANG modes (afl-gcc/afl-clang) have no README as they have no own 96 features 97 98You can select the mode for the afl-cc compiler by one of the following methods: 99 100* Using a symlink to afl-cc: afl-gcc, afl-g++, afl-clang, afl-clang++, 101 afl-clang-fast, afl-clang-fast++, afl-clang-lto, afl-clang-lto++, 102 afl-gcc-fast, afl-g++-fast (recommended!). 103* Using the environment variable `AFL_CC_COMPILER` with `MODE`. 104* Passing --afl-`MODE` command line options to the compiler via 105 `CFLAGS`/`CXXFLAGS`/`CPPFLAGS`. 106 107`MODE` can be one of the following: 108 109* LTO (afl-clang-lto*) 110* LLVM (afl-clang-fast*) 111* GCC_PLUGIN (afl-g*-fast) or GCC (afl-gcc/afl-g++) 112* CLANG(afl-clang/afl-clang++) 113 114Because no AFL++ specific command-line options are accepted (beside the 115--afl-MODE command), the compile-time tools make fairly broad use of environment 116variables, which can be listed with `afl-cc -hh` or looked up in 117[env_variables.md](env_variables.md). 118 119### b) Selecting instrumentation options 120 121If you instrument with LTO mode (afl-clang-fast/afl-clang-lto), the following 122options are available: 123 124* Splitting integer, string, float, and switch comparisons so AFL++ can easier 125 solve these. This is an important option if you do not have a very good and 126 large input corpus. This technique is called laf-intel or COMPCOV. To use 127 this, set the following environment variable before compiling the target: 128 `export AFL_LLVM_LAF_ALL=1`. You can read more about this in 129 [instrumentation/README.laf-intel.md](../instrumentation/README.laf-intel.md). 130* A different technique (and usually a better one than laf-intel) is to 131 instrument the target so that any compare values in the target are sent to 132 AFL++ which then tries to put these values into the fuzzing data at different 133 locations. This technique is very fast and good - if the target does not 134 transform input data before comparison. Therefore, this technique is called 135 `input to state` or `redqueen`. If you want to use this technique, then you 136 have to compile the target twice, once specifically with/for this mode by 137 setting `AFL_LLVM_CMPLOG=1`, and pass this binary to afl-fuzz via the `-c` 138 parameter. Note that you can compile also just a cmplog binary and use that 139 for both, however, there will be a performance penalty. You can read more 140 about this in 141 [instrumentation/README.cmplog.md](../instrumentation/README.cmplog.md). 142 143If you use LTO, LLVM, or GCC_PLUGIN mode 144(afl-clang-fast/afl-clang-lto/afl-gcc-fast), you have the option to selectively 145instrument _parts_ of the target that you are interested in. For afl-clang-fast, 146you have to use an llvm version newer than 10.0.0 or a mode other than 147DEFAULT/PCGUARD. 148 149This step can be done either by explicitly including parts to be instrumented or 150by explicitly excluding parts from instrumentation. 151 152* To instrument _only specified parts_, create a file (e.g., `allowlist.txt`) 153 with all the filenames and/or functions of the source code that should be 154 instrumented and then: 155 156 1. Just put one filename or function (prefixing with `fun: `) per line (no 157 directory information necessary for filenames) in the file `allowlist.txt`. 158 159 Example: 160 161 ``` 162 foo.cpp # will match foo/foo.cpp, bar/foo.cpp, barfoo.cpp etc. 163 fun: foo_func # will match the function foo_func 164 ``` 165 166 2. Set `export AFL_LLVM_ALLOWLIST=allowlist.txt` to enable selective positive 167 instrumentation. 168 169* Similarly to _exclude_ specified parts from instrumentation, create a file 170 (e.g., `denylist.txt`) with all the filenames of the source code that should 171 be skipped during instrumentation and then: 172 173 1. Same as above. Just put one filename or function per line in the file 174 `denylist.txt`. 175 176 2. Set `export AFL_LLVM_DENYLIST=denylist.txt` to enable selective negative 177 instrumentation. 178 179**NOTE:** During optimization functions might be 180inlined and then would not match the list! See 181[instrumentation/README.instrument_list.md](../instrumentation/README.instrument_list.md). 182 183There are many more options and modes available, however, these are most of the 184time less effective. See: 185 186* [instrumentation/README.llvm.md#6) AFL++ Context Sensitive Branch Coverage](../instrumentation/README.llvm.md#6-afl-context-sensitive-branch-coverage) 187* [instrumentation/README.llvm.md#7) AFL++ N-Gram Branch Coverage](../instrumentation/README.llvm.md#7-afl-n-gram-branch-coverage) 188 189AFL++ performs "never zero" counting in its bitmap. You can read more about this 190here: 191* [instrumentation/README.llvm.md#8-neverzero-counters](../instrumentation/README.llvm.md#8-neverzero-counters) 192 193### c) Selecting sanitizers 194 195It is possible to use sanitizers when instrumenting targets for fuzzing, which 196allows you to find bugs that would not necessarily result in a crash. 197 198Note that sanitizers have a huge impact on CPU (= less executions per second) 199and RAM usage. Also, you should only run one afl-fuzz instance per sanitizer 200type. This is enough because e.g. a use-after-free bug will be picked up by ASAN 201(address sanitizer) anyway after syncing test cases from other fuzzing 202instances, so running more than one address sanitized target would be a waste. 203 204The following sanitizers have built-in support in AFL++: 205 206* ASAN = Address SANitizer, finds memory corruption vulnerabilities like 207 use-after-free, NULL pointer dereference, buffer overruns, etc. Enabled with 208 `export AFL_USE_ASAN=1` before compiling. 209* MSAN = Memory SANitizer, finds read accesses to uninitialized memory, e.g., a 210 local variable that is defined and read before it is even set. Enabled with 211 `export AFL_USE_MSAN=1` before compiling. 212* UBSAN = Undefined Behavior SANitizer, finds instances where - by the C and C++ 213 standards - undefined behavior happens, e.g., adding two signed integers where 214 the result is larger than what a signed integer can hold. Enabled with `export 215 AFL_USE_UBSAN=1` before compiling. 216* CFISAN = Control Flow Integrity SANitizer, finds instances where the control 217 flow is found to be illegal. Originally this was rather to prevent return 218 oriented programming (ROP) exploit chains from functioning. In fuzzing, this 219 is mostly reduced to detecting type confusion vulnerabilities - which is, 220 however, one of the most important and dangerous C++ memory corruption 221 classes! Enabled with `export AFL_USE_CFISAN=1` before compiling. 222* TSAN = Thread SANitizer, finds thread race conditions. Enabled with `export 223 AFL_USE_TSAN=1` before compiling. 224* LSAN = Leak SANitizer, finds memory leaks in a program. This is not really a 225 security issue, but for developers this can be very valuable. Note that unlike 226 the other sanitizers above this needs `__AFL_LEAK_CHECK();` added to all areas 227 of the target source code where you find a leak check necessary! Enabled with 228 `export AFL_USE_LSAN=1` before compiling. To ignore the memory-leaking check 229 for certain allocations, `__AFL_LSAN_OFF();` can be used before memory is 230 allocated, and `__AFL_LSAN_ON();` afterwards. Memory allocated between these 231 two macros will not be checked for memory leaks. 232 233It is possible to further modify the behavior of the sanitizers at run-time by 234setting `ASAN_OPTIONS=...`, `LSAN_OPTIONS` etc. - the available parameters can 235be looked up in the sanitizer documentation of llvm/clang. afl-fuzz, however, 236requires some specific parameters important for fuzzing to be set. If you want 237to set your own, it might bail and report what it is missing. 238 239Note that some sanitizers cannot be used together, e.g., ASAN and MSAN, and 240others often cannot work together because of target weirdness, e.g., ASAN and 241CFISAN. You might need to experiment which sanitizers you can combine in a 242target (which means more instances can be run without a sanitized target, which 243is more effective). 244 245### d) Modifying the target 246 247If the target has features that make fuzzing more difficult, e.g., checksums, 248HMAC, etc., then modify the source code so that checks for these values are 249removed. This can even be done safely for source code used in operational 250products by eliminating these checks within these AFL++ specific blocks: 251 252``` 253#ifdef FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION 254 // say that the checksum or HMAC was fine - or whatever is required 255 // to eliminate the need for the fuzzer to guess the right checksum 256 return 0; 257#endif 258``` 259 260All AFL++ compilers will set this preprocessor definition automatically. 261 262### e) Instrumenting the target 263 264In this step, the target source code is compiled so that it can be fuzzed. 265 266Basically, you have to tell the target build system that the selected AFL++ 267compiler is used. Also - if possible - you should always configure the build 268system in such way that the target is compiled statically and not dynamically. 269How to do this is described below. 270 271The #1 rule when instrumenting a target is: avoid instrumenting shared libraries 272at all cost. You would need to set `LD_LIBRARY_PATH` to point to these, you 273could accidentally type "make install" and install them system wide - so don't. 274Really don't. **Always compile libraries you want to have instrumented as static 275and link these to the target program!** 276 277Then build the target. (Usually with `make`.) 278 279**NOTES** 280 2811. Sometimes configure and build systems are fickle and do not like stderr 282 output (and think this means a test failure) - which is something AFL++ likes 283 to do to show statistics. It is recommended to disable AFL++ instrumentation 284 reporting via `export AFL_QUIET=1`. 285 2862. Sometimes configure and build systems error on warnings - these should be 287 disabled (e.g., `--disable-werror` for some configure scripts). 288 2893. In case the configure/build system complains about AFL++'s compiler and 290 aborts, then set `export AFL_NOOPT=1` which will then just behave like the 291 real compiler and run the configure step separately. For building the target 292 afterwards this option has to be unset again! 293 294#### configure 295 296For `configure` build systems, this is usually done by: 297 298``` 299CC=afl-clang-fast CXX=afl-clang-fast++ ./configure --disable-shared 300``` 301 302Note that if you are using the (better) afl-clang-lto compiler, you also have to 303set `AR` to llvm-ar[-VERSION] and `RANLIB` to llvm-ranlib[-VERSION] - as is 304described in [instrumentation/README.lto.md](../instrumentation/README.lto.md). 305 306#### CMake 307 308For CMake build systems, this is usually done by: 309 310``` 311mkdir build; cd build; cmake -DCMAKE_C_COMPILER=afl-cc -DCMAKE_CXX_COMPILER=afl-c++ .. 312``` 313 314Note that if you are using the (better) afl-clang-lto compiler you also have to 315set AR to llvm-ar[-VERSION] and RANLIB to llvm-ranlib[-VERSION] - as is 316described in [instrumentation/README.lto.md](../instrumentation/README.lto.md). 317 318#### Meson Build System 319 320For the Meson Build System, you have to set the AFL++ compiler with the very 321first command! 322 323``` 324CC=afl-cc CXX=afl-c++ meson 325``` 326 327#### Other build systems or if configure/cmake didn't work 328 329Sometimes `cmake` and `configure` do not pick up the AFL++ compiler or the 330`RANLIB`/`AR` that is needed - because this was just not foreseen by the 331developer of the target. Or they have non-standard options. Figure out if there 332is a non-standard way to set this, otherwise set up the build normally and edit 333the generated build environment afterwards manually to point it to the right 334compiler (and/or `RANLIB` and `AR`). 335 336In complex, weird, alien build systems you can try this neat project: 337[https://github.com/fuzzah/exeptor](https://github.com/fuzzah/exeptor) 338 339#### Linker scripts 340 341If the project uses linker scripts to hide the symbols exported by the 342binary, then you may see errors such as: 343 344``` 345undefined symbol: __afl_area_ptr 346``` 347 348The solution is to modify the linker script to add: 349 350``` 351{ 352 global: 353 __afl_*; 354} 355``` 356 357### f) Better instrumentation 358 359If you just fuzz a target program as-is, you are wasting a great opportunity for 360much more fuzzing speed. 361 362This variant requires the usage of afl-clang-lto, afl-clang-fast or 363afl-gcc-fast. 364 365It is the so-called `persistent mode`, which is much, much faster but requires 366that you code a source file that is specifically calling the target functions 367that you want to fuzz, plus a few specific AFL++ functions around it. See 368[instrumentation/README.persistent_mode.md](../instrumentation/README.persistent_mode.md) 369for details. 370 371Basically, if you do not fuzz a target in persistent mode, then you are just 372doing it for a hobby and not professionally :-). 373 374### g) libfuzzer fuzzer harnesses with LLVMFuzzerTestOneInput() 375 376libfuzzer `LLVMFuzzerTestOneInput()` harnesses are the defacto standard for 377fuzzing, and they can be used with AFL++ (and honggfuzz) as well! 378 379Compiling them is as simple as: 380 381``` 382afl-clang-fast++ -fsanitize=fuzzer -o harness harness.cpp targetlib.a 383``` 384 385You can even use advanced libfuzzer features like `FuzzedDataProvider`, 386`LLVMFuzzerInitialize()` etc. and they will work! 387 388The generated binary is fuzzed with afl-fuzz like any other fuzz target. 389 390Bonus: the target is already optimized for fuzzing due to persistent mode and 391shared-memory test cases and hence gives you the fastest speed possible. 392 393For more information, see 394[utils/aflpp_driver/README.md](../utils/aflpp_driver/README.md). 395 396## 2. Preparing the fuzzing campaign 397 398As you fuzz the target with mutated input, having as diverse inputs for the 399target as possible improves the efficiency a lot. 400 401### a) Collecting inputs 402 403To operate correctly, the fuzzer requires one or more starting files that 404contain a good example of the input data normally expected by the targeted 405application. 406 407Try to gather valid inputs for the target from wherever you can. E.g., if it is 408the PNG picture format, try to find as many PNG files as possible, e.g., from 409reported bugs, test suites, random downloads from the internet, unit test case 410data - from all kind of PNG software. 411 412If the input format is not known, you can also modify a target program to write 413normal data it receives and processes to a file and use these. 414 415You can find many good examples of starting files in the 416[testcases/](../testcases) subdirectory that comes with this tool. 417 418### b) Making the input corpus unique 419 420Use the AFL++ tool `afl-cmin` to remove inputs from the corpus that do not 421produce a new path/coverage in the target: 422 4231. Put all files from [step a](#a-collecting-inputs) into one directory, e.g., 424 `INPUTS`. 4252. Run afl-cmin: 426 * If the target program is to be called by fuzzing as `bin/target INPUTFILE`, 427 replace the INPUTFILE argument that the target program would read from with 428 `@@`: 429 430 ``` 431 afl-cmin -i INPUTS -o INPUTS_UNIQUE -- bin/target -someopt @@ 432 ``` 433 434 * If the target reads from stdin (standard input) instead, just omit the `@@` 435 as this is the default: 436 437 ``` 438 afl-cmin -i INPUTS -o INPUTS_UNIQUE -- bin/target -someopt 439 ``` 440 441This step is highly recommended, because afterwards the testcase corpus is not 442bloated with duplicates anymore, which would slow down the fuzzing progress! 443 444### c) Minimizing all corpus files 445 446The shorter the input files that still traverse the same path within the target, 447the better the fuzzing will be. This minimization is done with `afl-tmin`, 448however, it is a long process as this has to be done for every file: 449 450``` 451mkdir input 452cd INPUTS_UNIQUE 453for i in *; do 454 afl-tmin -i "$i" -o "../input/$i" -- bin/target -someopt @@ 455done 456``` 457 458This step can also be parallelized, e.g., with `parallel`. 459 460Note that this step is rather optional though. 461 462### Done! 463 464The INPUTS_UNIQUE/ directory from [step b](#b-making-the-input-corpus-unique) - 465or even better the directory input/ if you minimized the corpus in 466[step c](#c-minimizing-all-corpus-files) - is the resulting input corpus 467directory to be used in fuzzing! :-) 468 469## 3. Fuzzing the target 470 471In this final step, fuzz the target. There are not that many important options 472to run the target - unless you want to use many CPU cores/threads for the 473fuzzing, which will make the fuzzing much more useful. 474 475If you just use one instance for fuzzing, then you are fuzzing just for fun and 476not seriously :-) 477 478### a) Running afl-fuzz 479 480Before you do even a test run of afl-fuzz, execute `sudo afl-system-config` (on 481the host if you execute afl-fuzz in a Docker container). This reconfigures the 482system for optimal speed - which afl-fuzz checks and bails otherwise. Set 483`export AFL_SKIP_CPUFREQ=1` for afl-fuzz to skip this check if you cannot run 484afl-system-config with root privileges on the host for whatever reason. 485 486Note: 487 488* There is also `sudo afl-persistent-config` which sets additional permanent 489 boot options for a much better fuzzing performance. 490* Both scripts improve your fuzzing performance but also decrease your system 491 protection against attacks! So set strong firewall rules and only expose SSH 492 as a network service if you use these (which is highly recommended). 493 494If you have an input corpus from [step 2](#2-preparing-the-fuzzing-campaign), 495then specify this directory with the `-i` option. Otherwise, create a new 496directory and create a file with any content as test data in there. 497 498If you do not want anything special, the defaults are already usually best, 499hence all you need is to specify the seed input directory with the result of 500step [2a) Collecting inputs](#a-collecting-inputs): 501 502``` 503afl-fuzz -i input -o output -- bin/target -someopt @@ 504``` 505 506Note that the directory specified with `-o` will be created if it does not 507exist. 508 509It can be valuable to run afl-fuzz in a `screen` or `tmux` shell so you can log 510off, or afl-fuzz is not aborted if you are running it in a remote ssh session 511where the connection fails in between. Only do that though once you have 512verified that your fuzzing setup works! Run it like `screen -dmS afl-main -- 513afl-fuzz -M main-$HOSTNAME -i ...` and it will start away in a screen session. 514To enter this session, type `screen -r afl-main`. You see - it makes sense to 515name the screen session same as the afl-fuzz `-M`/`-S` naming :-) For more 516information on screen or tmux, check their documentation. 517 518If you need to stop and re-start the fuzzing, use the same command line options 519(or even change them by selecting a different power schedule or another mutation 520mode!) and switch the input directory with a dash (`-`): 521 522``` 523afl-fuzz -i - -o output -- bin/target -someopt @@ 524``` 525 526Adding a dictionary is helpful. You have the following options: 527 528* See the directory 529[dictionaries/](../dictionaries/), if something is already included for your 530data format, and tell afl-fuzz to load that dictionary by adding `-x 531dictionaries/FORMAT.dict`. 532* With `afl-clang-lto`, you have an autodictionary generation for which you need 533 to do nothing except to use afl-clang-lto as the compiler. 534* With `afl-clang-fast`, you can set 535 `AFL_LLVM_DICT2FILE=/full/path/to/new/file.dic` to automatically generate a 536 dictionary during target compilation. 537 Adding `AFL_LLVM_DICT2FILE_NO_MAIN=1` to not parse main (usually command line 538 parameter parsing) is often a good idea too. 539* You also have the option to generate a dictionary yourself during an 540 independent run of the target, see 541 [utils/libtokencap/README.md](../utils/libtokencap/README.md). 542* Finally, you can also write a dictionary file manually, of course. 543 544afl-fuzz has a variety of options that help to workaround target quirks like 545very specific locations for the input file (`-f`), performing deterministic 546fuzzing (`-D`) and many more. Check out `afl-fuzz -h`. 547 548We highly recommend that you set a memory limit for running the target with `-m` 549which defines the maximum memory in MB. This prevents a potential out-of-memory 550problem for your system plus helps you detect missing `malloc()` failure 551handling in the target. Play around with various `-m` values until you find one 552that safely works for all your input seeds (if you have good ones and then 553double or quadruple that). 554 555By default, afl-fuzz never stops fuzzing. To terminate AFL++, press Control-C or 556send a signal SIGINT. You can limit the number of executions or approximate 557runtime in seconds with options also. 558 559When you start afl-fuzz, you will see a user interface that shows what the 560status is: 561 562 563 564All labels are explained in 565[afl-fuzz_approach.md#understanding-the-status-screen](afl-fuzz_approach.md#understanding-the-status-screen). 566 567### b) Keeping memory use and timeouts in check 568 569Memory limits are not enforced by afl-fuzz by default and the system may run out 570of memory. You can decrease the memory with the `-m` option, the value is in MB. 571If this is too small for the target, you can usually see this by afl-fuzz 572bailing with the message that it could not connect to the forkserver. 573 574Consider setting low values for `-m` and `-t`. 575 576For programs that are nominally very fast, but get sluggish for some inputs, you 577can also try setting `-t` values that are more punishing than what `afl-fuzz` 578dares to use on its own. On fast and idle machines, going down to `-t 5` may be 579a viable plan. 580 581The `-m` parameter is worth looking at, too. Some programs can end up spending a 582fair amount of time allocating and initializing megabytes of memory when 583presented with pathological inputs. Low `-m` values can make them give up sooner 584and not waste CPU time. 585 586### c) Using multiple cores 587 588If you want to seriously fuzz, then use as many cores/threads as possible to 589fuzz your target. 590 591On the same machine - due to the design of how AFL++ works - there is a maximum 592number of CPU cores/threads that are useful, use more and the overall 593performance degrades instead. This value depends on the target, and the limit is 594between 32 and 64 cores per machine. 595 596If you have the RAM, it is highly recommended run the instances with a caching 597of the test cases. Depending on the average test case size (and those found 598during fuzzing) and their number, a value between 50-500MB is recommended. You 599can set the cache size (in MB) by setting the environment variable 600`AFL_TESTCACHE_SIZE`. 601 602There should be one main fuzzer (`-M main-$HOSTNAME` option - set also 603`AFL_FINAL_SYNC=1`) and as many secondary fuzzers (e.g., `-S variant1`) as you 604have cores that you use. Every `-M`/`-S` entry needs a unique name (that can be 605whatever), however, the same `-o` output directory location has to be used for 606all instances. 607 608For every secondary fuzzer there should be a variation, e.g.: 609* one should fuzz the target that was compiled with sanitizers activated 610 (`export AFL_USE_ASAN=1 ; export AFL_USE_UBSAN=1 ; export AFL_USE_CFISAN=1`) 611* one or two should fuzz the target with CMPLOG/redqueen (see above), at least 612 one cmplog instance should follow transformations (`-l 2AT`) 613* one to three fuzzers should fuzz a target compiled with laf-intel/COMPCOV (see 614 above). Important note: If you run more than one laf-intel/COMPCOV fuzzer and 615 you want them to share their intermediate results, the main fuzzer (`-M`) must 616 be one of them (although this is not really recommended). 617 618The other secondaries should be run like this: 619* 10% with the MOpt mutator enabled: `-L 0` 620* 10% should use the old queue cycling with `-Z` 621* 50-70% should run with `AFL_DISABLE_TRIM` 622* 40% should run with `-P explore` and 20% with `-P exploit` 623* If you use `-a` then set 30% of the instances to not use `-a`; if you did 624 not set `-a` (why??), then set 30% to `-a ascii` and 30% to `-a binary`. 625* run each with a different power schedule, recommended are: `fast` (default), 626 `explore`, `coe`, `lin`, `quad`, `exploit`, and `rare` which you can set with 627 the `-p` option, e.g., `-p explore`. See the 628 [FAQ](FAQ.md#what-are-power-schedules) for details. 629 630It can be useful to set `AFL_IGNORE_SEED_PROBLEMS=1` to skip over seeds that 631crash or timeout during startup. 632 633Also, it is recommended to set `export AFL_IMPORT_FIRST=1` to load test cases 634from other fuzzers in the campaign first. But note that can slow down the start 635of the first fuzz by quite a lot of you have many fuzzers and/or many seeds. 636 637If you have a large corpus, a corpus from a previous run or are fuzzing in a CI, 638then also set `export AFL_CMPLOG_ONLY_NEW=1` and `export AFL_FAST_CAL=1`. 639If the queue in the CI is huge and/or the execution time is slow then you can 640also add `AFL_NO_STARTUP_CALIBRATION=1` to skip the initial queue calibration 641phase and start fuzzing at once - but only do this if the calibration phase 642would be too long for your fuzz run time. 643 644You can also use different fuzzers. If you are using AFL spinoffs or AFL 645conforming fuzzers, then just use the same -o directory and give it a unique 646`-S` name. Examples are: 647* [Fuzzolic](https://github.com/season-lab/fuzzolic) 648* [symcc](https://github.com/eurecom-s3/symcc/) 649* [Eclipser](https://github.com/SoftSec-KAIST/Eclipser/) 650* [AFLsmart](https://github.com/aflsmart/aflsmart) 651* [FairFuzz](https://github.com/carolemieux/afl-rb) 652* [Neuzz](https://github.com/Dongdongshe/neuzz) 653* [Angora](https://github.com/AngoraFuzzer/Angora) 654 655A long list can be found at 656[https://github.com/Microsvuln/Awesome-AFL](https://github.com/Microsvuln/Awesome-AFL). 657 658However, you can also sync AFL++ with honggfuzz, libfuzzer with `-entropic=1`, 659etc. Just show the main fuzzer (`-M`) with the `-F` option where the queue/work 660directory of a different fuzzer is, e.g., `-F /src/target/honggfuzz`. Using 661honggfuzz (with `-n 1` or `-n 2`) and libfuzzer in parallel is highly 662recommended! 663 664### d) Using multiple machines for fuzzing 665 666Maybe you have more than one machine you want to fuzz the same target on. Start 667the `afl-fuzz` (and perhaps libfuzzer, honggfuzz, ...) orchestra as you like, 668just ensure that your have one and only one `-M` instance per server, and that 669its name is unique, hence the recommendation for `-M main-$HOSTNAME`. 670 671Now there are three strategies on how you can sync between the servers: 672* never: sounds weird, but this makes every server an island and has the chance 673 that each follow different paths into the target. You can make this even more 674 interesting by even giving different seeds to each server. 675* regularly (~4h): this ensures that all fuzzing campaigns on the servers "see" 676 the same thing. It is like fuzzing on a huge server. 677* in intervals of 1/10th of the overall expected runtime of the fuzzing you 678 sync. This tries a bit to combine both. Have some individuality of the paths 679 each campaign on a server explores, on the other hand if one gets stuck where 680 another found progress this is handed over making it unstuck. 681 682The syncing process itself is very simple. As the `-M main-$HOSTNAME` instance 683syncs to all `-S` secondaries as well as to other fuzzers, you have to copy only 684this directory to the other machines. 685 686Let's say all servers have the `-o out` directory in /target/foo/out, and you 687created a file `servers.txt` which contains the hostnames of all participating 688servers, plus you have an ssh key deployed to all of them, then run: 689 690```bash 691for FROM in `cat servers.txt`; do 692 for TO in `cat servers.txt`; do 693 rsync -rlpogtz --rsh=ssh $FROM:/target/foo/out/main-$FROM $TO:target/foo/out/ 694 done 695done 696``` 697 698You can run this manually, per cron job - as you need it. There is a more 699complex and configurable script in 700[utils/distributed_fuzzing](../utils/distributed_fuzzing). 701 702### e) The status of the fuzz campaign 703 704AFL++ comes with the `afl-whatsup` script to show the status of the fuzzing 705campaign. 706 707Just supply the directory that afl-fuzz is given with the `-o` option and you 708will see a detailed status of every fuzzer in that campaign plus a summary. 709 710To have only the summary, use the `-s` switch, e.g., `afl-whatsup -s out/`. 711 712If you have multiple servers, then use the command after a sync or you have to 713execute this script per server. 714 715Another tool to inspect the current state and history of a specific instance is 716afl-plot, which generates an index.html file and graphs that show how the 717fuzzing instance is performing. The syntax is `afl-plot instance_dir web_dir`, 718e.g., `afl-plot out/default /srv/www/htdocs/plot`. 719 720### f) Stopping fuzzing, restarting fuzzing, adding new seeds 721 722To stop an afl-fuzz run, press Control-C. 723 724To restart an afl-fuzz run, just reuse the same command line but replace the `-i 725directory` with `-i -` or set `AFL_AUTORESUME=1`. 726 727If you want to add new seeds to a fuzzing campaign, you can run a temporary 728fuzzing instance, e.g., when your main fuzzer is using `-o out` and the new 729seeds are in `newseeds/` directory: 730 731``` 732AFL_BENCH_JUST_ONE=1 AFL_FAST_CAL=1 afl-fuzz -i newseeds -o out -S newseeds -- ./target 733``` 734 735### g) Checking the coverage of the fuzzing 736 737The `corpus count` value is a bad indicator for checking how good the coverage 738is. 739 740A better indicator - if you use default llvm instrumentation with at least 741version 9 - is to use `afl-showmap` with the collect coverage option `-C` on the 742output directory: 743 744``` 745$ afl-showmap -C -i out -o /dev/null -- ./target -params @@ 746... 747[*] Using SHARED MEMORY FUZZING feature. 748[*] Target map size: 9960 749[+] Processed 7849 input files. 750[+] Captured 4331 tuples (highest value 255, total values 67130596) in '/dev/nul 751l'. 752[+] A coverage of 4331 edges were achieved out of 9960 existing (43.48%) with 7849 input files. 753``` 754 755It is even better to check out the exact lines of code that have been reached - 756and which have not been found so far. 757 758An "easy" helper script for this is 759[https://github.com/vanhauser-thc/afl-cov](https://github.com/vanhauser-thc/afl-cov), 760just follow the README of that separate project. 761 762If you see that an important area or a feature has not been covered so far, then 763try to find an input that is able to reach that and start a new secondary in 764that fuzzing campaign with that seed as input, let it run for a few minutes, 765then terminate it. The main node will pick it up and make it available to the 766other secondary nodes over time. Set `export AFL_NO_AFFINITY=1` or `export 767AFL_TRY_AFFINITY=1` if you have no free core. 768 769Note that in nearly all cases you can never reach full coverage. A lot of 770functionality is usually dependent on exclusive options that would need 771individual fuzzing campaigns each with one of these options set. E.g., if you 772fuzz a library to convert image formats and your target is the png to tiff API, 773then you will not touch any of the other library APIs and features. 774 775### h) How long to fuzz a target? 776 777This is a difficult question. Basically, if no new path is found for a long time 778(e.g., for a day or a week), then you can expect that your fuzzing won't be 779fruitful anymore. However, often this just means that you should switch out 780secondaries for others, e.g., custom mutator modules, sync to very different 781fuzzers, etc. 782 783Keep the queue/ directory (for future fuzzings of the same or similar targets) 784and use them to seed other good fuzzers like libfuzzer with the -entropic switch 785or honggfuzz. 786 787### i) Improve the speed! 788 789* Use [persistent mode](../instrumentation/README.persistent_mode.md) (x2-x20 790 speed increase). 791* If you do not use shmem persistent mode, use `AFL_TMPDIR` to point the input 792 file on a tempfs location, see [env_variables.md](env_variables.md). 793* Linux: Improve kernel performance: modify `/etc/default/grub`, set 794 `GRUB_CMDLINE_LINUX_DEFAULT="ibpb=off ibrs=off kpti=off l1tf=off mds=off 795 mitigations=off no_stf_barrier noibpb noibrs nopcid nopti 796 nospec_store_bypass_disable nospectre_v1 nospectre_v2 pcid=off pti=off 797 spec_store_bypass_disable=off spectre_v2=off stf_barrier=off"`; then 798 `update-grub` and `reboot` (warning: makes the system more insecure) - you can 799 also just run `sudo afl-persistent-config`. 800* Linux: Running on an `ext2` filesystem with `noatime` mount option will be a 801 bit faster than on any other journaling filesystem. 802* Use your cores! See [3c) Using multiple cores](#c-using-multiple-cores). 803* Run `sudo afl-system-config` before starting the first afl-fuzz instance after 804 a reboot. 805 806### j) Going beyond crashes 807 808Fuzzing is a wonderful and underutilized technique for discovering non-crashing 809design and implementation errors, too. Quite a few interesting bugs have been 810found by modifying the target programs to call `abort()` when say: 811 812- Two bignum libraries produce different outputs when given the same 813 fuzzer-generated input. 814 815- An image library produces different outputs when asked to decode the same 816 input image several times in a row. 817 818- A serialization/deserialization library fails to produce stable outputs when 819 iteratively serializing and deserializing fuzzer-supplied data. 820 821- A compression library produces an output inconsistent with the input file when 822 asked to compress and then decompress a particular blob. 823 824Implementing these or similar sanity checks usually takes very little time; if 825you are the maintainer of a particular package, you can make this code 826conditional with `#ifdef FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION` (a flag also 827shared with libfuzzer and honggfuzz) or `#ifdef __AFL_COMPILER` (this one is 828just for AFL++). 829 830### k) Known limitations & areas for improvement 831 832Here are some of the most important caveats for AFL++: 833 834- AFL++ detects faults by checking for the first spawned process dying due to a 835 signal (SIGSEGV, SIGABRT, etc.). Programs that install custom handlers for 836 these signals may need to have the relevant code commented out. In the same 837 vein, faults in child processes spawned by the fuzzed target may evade 838 detection unless you manually add some code to catch that. 839 840- As with any other brute-force tool, the fuzzer offers limited coverage if 841 encryption, checksums, cryptographic signatures, or compression are used to 842 wholly wrap the actual data format to be tested. 843 844 To work around this, you can comment out the relevant checks (see 845 utils/libpng_no_checksum/ for inspiration); if this is not possible, you can 846 also write a postprocessor, one of the hooks of custom mutators. See 847 [custom_mutators.md](custom_mutators.md) on how to use 848 `AFL_CUSTOM_MUTATOR_LIBRARY`. 849 850- There are some unfortunate trade-offs with ASAN and 64-bit binaries. This 851 isn't due to any specific fault of afl-fuzz. 852 853- There is no direct support for fuzzing network services, background daemons, 854 or interactive apps that require UI interaction to work. You may need to make 855 simple code changes to make them behave in a more traditional way. Preeny or 856 libdesock may offer a relatively simple option, too - see: 857 [https://github.com/zardus/preeny](https://github.com/zardus/preeny) or 858 [https://github.com/fkie-cad/libdesock](https://github.com/fkie-cad/libdesock) 859 860 Some useful tips for modifying network-based services can be also found at: 861 [https://www.fastly.com/blog/how-to-fuzz-server-american-fuzzy-lop](https://www.fastly.com/blog/how-to-fuzz-server-american-fuzzy-lop) 862 863- Occasionally, sentient machines rise against their creators. If this happens 864 to you, please consult 865 [https://lcamtuf.coredump.cx/prep/](https://lcamtuf.coredump.cx/prep/). 866 867Beyond this, see [INSTALL.md](INSTALL.md) for platform-specific tips. 868 869## 4. Triaging crashes 870 871The coverage-based grouping of crashes usually produces a small data set that 872can be quickly triaged manually or with a very simple GDB or Valgrind script. 873Every crash is also traceable to its parent non-crashing test case in the queue, 874making it easier to diagnose faults. 875 876Having said that, it's important to acknowledge that some fuzzing crashes can be 877difficult to quickly evaluate for exploitability without a lot of debugging and 878code analysis work. To assist with this task, afl-fuzz supports a very unique 879"crash exploration" mode enabled with the `-C` flag. 880 881In this mode, the fuzzer takes one or more crashing test cases as the input and 882uses its feedback-driven fuzzing strategies to very quickly enumerate all code 883paths that can be reached in the program while keeping it in the crashing state. 884 885Mutations that do not result in a crash are rejected; so are any changes that do 886not affect the execution path. 887 888The output is a small corpus of files that can be very rapidly examined to see 889what degree of control the attacker has over the faulting address, or whether it 890is possible to get past an initial out-of-bounds read - and see what lies 891beneath. 892 893Oh, one more thing: for test case minimization, give afl-tmin a try. The tool 894can be operated in a very simple way: 895 896```shell 897./afl-tmin -i test_case -o minimized_result -- /path/to/program [...] 898``` 899 900The tool works with crashing and non-crashing test cases alike. In the crash 901mode, it will happily accept instrumented and non-instrumented binaries. In the 902non-crashing mode, the minimizer relies on standard AFL++ instrumentation to 903make the file simpler without altering the execution path. 904 905The minimizer accepts the `-m`, `-t`, `-f`, and `@@` syntax in a manner 906compatible with afl-fuzz. 907 908Another tool in AFL++ is the afl-analyze tool. It takes an input file, attempts 909to sequentially flip bytes and observes the behavior of the tested program. It 910then color-codes the input based on which sections appear to be critical and 911which are not; while not bulletproof, it can often offer quick insights into 912complex file formats. 913 914`casr-afl` from [CASR](https://github.com/ispras/casr) tools provides 915comfortable triaging for crashes found by AFL++. Reports are clustered and 916contain severity and other information. 917```shell 918casr-afl -i /path/to/afl/out/dir -o /path/to/casr/out/dir 919``` 920 921## 5. CI fuzzing 922 923Some notes on continuous integration (CI) fuzzing - this fuzzing is different to 924normal fuzzing campaigns as these are much shorter runnings. 925 926If the queue in the CI is huge and/or the execution time is slow then you can 927also add `AFL_NO_STARTUP_CALIBRATION=1` to skip the initial queue calibration 928phase and start fuzzing at once. But only do that if the calibration time is 929too long for your overall available fuzz run time. 930 9311. Always: 932 * LTO has a much longer compile time which is diametrical to short fuzzing - 933 hence use afl-clang-fast instead. 934 * If you compile with CMPLOG, then you can save compilation time and reuse 935 that compiled target with the `-c` option and as the main fuzz target. 936 This will impact the speed by ~15% though. 937 * `AFL_FAST_CAL` - enables fast calibration, this halves the time the 938 saturated corpus needs to be loaded. 939 * `AFL_CMPLOG_ONLY_NEW` - only perform cmplog on new finds, not the initial 940 corpus as this very likely has been done for them already. 941 * Keep the generated corpus, use afl-cmin and reuse it every time! 942 9432. Additionally randomize the AFL++ compilation options, e.g.: 944 * 30% for `AFL_LLVM_CMPLOG` 945 * 5% for `AFL_LLVM_LAF_ALL` 946 9473. Also randomize the afl-fuzz runtime options, e.g.: 948 * 65% for `AFL_DISABLE_TRIM` 949 * 50% for `AFL_KEEP_TIMEOUTS` 950 * 50% use a dictionary generated by `AFL_LLVM_DICT2FILE` + `AFL_LLVM_DICT2FILE_NO_MAIN=1` 951 * 10% use MOpt (`-L 0`) 952 * 40% for `AFL_EXPAND_HAVOC_NOW` 953 * 20% for old queue processing (`-Z`) 954 * for CMPLOG targets, 70% for `-l 2`, 10% for `-l 3`, 20% for `-l 2AT` 955 9564. Do *not* run any `-M` modes, just running `-S` modes is better for CI 957 fuzzing. `-M` enables old queue handling etc. which is good for a fuzzing 958 campaign but not good for short CI runs. 959 960How this can look like can, e.g., be seen at AFL++'s setup in Google's 961[oss-fuzz](https://github.com/google/oss-fuzz/blob/master/infra/base-images/base-builder/compile_afl) 962and 963[clusterfuzz](https://github.com/google/clusterfuzz/blob/master/src/clusterfuzz/_internal/bot/fuzzers/afl/launcher.py). 964 965## The End 966 967Check out the [FAQ](FAQ.md). Maybe it answers your question (that you might not 968even have known you had ;-) ). 969 970This is basically all you need to know to professionally run fuzzing campaigns. 971If you want to know more, the tons of texts in [docs/](./) will have you 972covered. 973 974Note that there are also a lot of tools out there that help fuzzing with AFL++ 975(some might be deprecated or unsupported), see 976[third_party_tools.md](third_party_tools.md). 977