1# Best practices 2 3## Contents 4 5### Targets 6 7* [Fuzzing a target with source code available](#fuzzing-a-target-with-source-code-available) 8* [Fuzzing a target with dlopen() instrumented libraries](#fuzzing-a-target-with-dlopen-instrumented-libraries) 9* [Fuzzing a binary-only target](#fuzzing-a-binary-only-target) 10* [Fuzzing a GUI program](#fuzzing-a-gui-program) 11* [Fuzzing a network service](#fuzzing-a-network-service) 12 13### Improvements 14 15* [Improving speed](#improving-speed) 16* [Improving stability](#improving-stability) 17 18## Targets 19 20### Fuzzing a target with source code available 21 22To learn how to fuzz a target if source code is available, see 23[fuzzing_in_depth.md](fuzzing_in_depth.md). 24 25### Fuzzing a target with dlopen instrumented libraries 26 27If a source code based fuzzing target loads instrumented libraries with 28dlopen() after the forkserver has been activated and non-colliding coverage 29instrumentation is used (PCGUARD (which is the default), or LTO), then this 30an issue, because this would enlarge the coverage map, but afl-fuzz doesn't 31know about it. 32 33The solution is to use `AFL_PRELOAD` for all dlopen()'ed libraries to 34ensure that all coverage targets are present on startup in the target, 35even if accessed only later with dlopen(). 36 37For PCGUARD instrumentation `abort()` is called if this is detected, for LTO 38there will either be no coverage for the instrumented dlopen()'ed libraries or 39you will see lots of crashes in the UI. 40 41Note that this is not an issue if you use the inferiour `afl-gcc-fast`, 42`afl-gcc` or`AFL_LLVM_INSTRUMENT=CLASSIC/NGRAM/CTX afl-clang-fast` 43instrumentation. 44 45### Fuzzing a binary-only target 46 47For a comprehensive guide, see 48[fuzzing_binary-only_targets.md](fuzzing_binary-only_targets.md). 49 50### Fuzzing a GUI program 51 52If the GUI program can read the fuzz data from a file (via the command line, a 53fixed location or via an environment variable) without needing any user 54interaction, then it would be suitable for fuzzing. 55 56Otherwise, it is not possible without modifying the source code - which is a 57very good idea anyway as the GUI functionality is a huge CPU/time overhead for 58the fuzzing. 59 60So create a new `main()` that just reads the test case and calls the 61functionality for processing the input that the GUI program is using. 62 63### Fuzzing a network service 64 65Fuzzing a network service does not work "out of the box". 66 67Using a network channel is inadequate for several reasons: 68- it has a slow-down of x10-20 on the fuzzing speed 69- it does not scale to fuzzing multiple instances easily, 70- instead of one initial data packet often a back-and-forth interplay of packets 71 is needed for stateful protocols (which is totally unsupported by most 72 coverage aware fuzzers). 73 74The established method to fuzz network services is to modify the source code to 75read from a file or stdin (fd 0) (or even faster via shared memory, combine this 76with persistent mode 77[instrumentation/README.persistent_mode.md](../instrumentation/README.persistent_mode.md) 78and you have a performance gain of x10 instead of a performance loss of over x10 79- that is a x100 difference!). 80 81If modifying the source is not an option (e.g., because you only have a binary 82and perform binary fuzzing) you can also use a shared library with AFL_PRELOAD 83to emulate the network. This is also much faster than the real network would be. 84See [utils/socket_fuzzing/](../utils/socket_fuzzing/). 85 86There is an outdated AFL++ branch that implements networking if you are 87desperate though: 88[https://github.com/AFLplusplus/AFLplusplus/tree/networking](https://github.com/AFLplusplus/AFLplusplus/tree/networking) 89- however, a better option is AFLnet 90([https://github.com/aflnet/aflnet](https://github.com/aflnet/aflnet)) which 91allows you to define network state with different type of data packets. 92 93## Improvements 94 95### Improving speed 96 971. Use [llvm_mode](../instrumentation/README.llvm.md): afl-clang-lto (llvm >= 98 11) or afl-clang-fast (llvm >= 9 recommended). 992. Use [persistent mode](../instrumentation/README.persistent_mode.md) (x2-x20 100 speed increase). 1013. Instrument just what you are interested in, see 102 [instrumentation/README.instrument_list.md](../instrumentation/README.instrument_list.md). 1034. If you do not use shmem persistent mode, use `AFL_TMPDIR` to put the input 104 file directory on a tempfs location, see 105 [env_variables.md](env_variables.md). 1065. Improve Linux kernel performance: modify `/etc/default/grub`, set 107 `GRUB_CMDLINE_LINUX_DEFAULT="ibpb=off ibrs=off kpti=off l1tf=off mds=off 108 mitigations=off no_stf_barrier noibpb noibrs nopcid nopti 109 nospec_store_bypass_disable nospectre_v1 nospectre_v2 pcid=off pti=off 110 spec_store_bypass_disable=off spectre_v2=off stf_barrier=off"`; then 111 `update-grub` and `reboot` (warning: makes the system less secure). 1126. Running on an `ext2` filesystem with `noatime` mount option will be a bit 113 faster than on any other journaling filesystem. 1147. Use your cores 115 ([fuzzing_in_depth.md:3c) Using multiple cores](fuzzing_in_depth.md#c-using-multiple-cores))! 116 117### Improving stability 118 119For fuzzing, a 100% stable target that covers all edges is the best case. A 90% 120stable target that covers all edges is, however, better than a 100% stable 121target that ignores 10% of the edges. 122 123With instability, you basically have a partial coverage loss on an edge, with 124ignored functions you have a full loss on that edges. 125 126There are functions that are unstable, but also provide value to coverage, e.g., 127init functions that use fuzz data as input. If, however, a function that has 128nothing to do with the input data is the source of instability, e.g., checking 129jitter, or is a hash map function etc., then it should not be instrumented. 130 131To be able to exclude these functions (based on AFL++'s measured stability), the 132following process will allow to identify functions with variable edges. 133 134Note that this is only useful for non-persistent targets! 135If a persistent target is unstable whereas when run non-persistent is fine, 136then this means that the target is keeping internal state, which is bad for 137fuzzing. Fuzz such targets **without** persistent mode. 138 139Four steps are required to do this and it also requires quite some knowledge of 140coding and/or disassembly and is effectively possible only with `afl-clang-fast` 141`PCGUARD` and `afl-clang-lto` `LTO` instrumentation. 142 143 1. Instrument to be able to find the responsible function(s): 144 145 a) For LTO instrumented binaries, this can be documented during compile 146 time, just set `export AFL_LLVM_DOCUMENT_IDS=/path/to/a/file`. This file 147 will have one assigned edge ID and the corresponding function per line. 148 149 b) For PCGUARD instrumented binaries, it is much more difficult. Here you 150 can either modify the `__sanitizer_cov_trace_pc_guard` function in 151 `instrumentation/afl-llvm-rt.o.c` to write a backtrace to a file if the 152 ID in `__afl_area_ptr[*guard]` is one of the unstable edge IDs. (Example 153 code is already there). Then recompile and reinstall `llvm_mode` and 154 rebuild your target. Run the recompiled target with `afl-fuzz` for a 155 while and then check the file that you wrote with the backtrace 156 information. Alternatively, you can use `gdb` to hook 157 `__sanitizer_cov_trace_pc_guard_init` on start, check to which memory 158 address the edge ID value is written, and set a write breakpoint to that 159 address (`watch 0x.....`). 160 161 c) In other instrumentation types, this is not possible. So just recompile 162 with the two mentioned above. This is just for identifying the functions 163 that have unstable edges. 164 165 2. Identify which edge ID numbers are unstable. 166 167 Run the target with `export AFL_DEBUG=1` for a few minutes then terminate. 168 The out/fuzzer_stats file will then show the edge IDs that were identified 169 as unstable in the `var_bytes` entry. You can match these numbers directly 170 to the data you created in the first step. Now you know which functions are 171 responsible for the instability 172 173 3. Create a text file with the filenames/functions 174 175 Identify which source code files contain the functions that you need to 176 remove from instrumentation, or just specify the functions you want to skip 177 for instrumentation. Note that optimization might inline functions! 178 179 Follow this document on how to do this: 180 [instrumentation/README.instrument_list.md](../instrumentation/README.instrument_list.md). 181 182 If `PCGUARD` is used, then you need to follow this guide (needs llvm 12+!): 183 [https://clang.llvm.org/docs/SanitizerCoverage.html#partially-disabling-instrumentation](https://clang.llvm.org/docs/SanitizerCoverage.html#partially-disabling-instrumentation) 184 185 Only exclude those functions from instrumentation that provide no value for 186 coverage - that is if it does not process any fuzz data directly or 187 indirectly (e.g., hash maps, thread management etc.). If, however, a 188 function directly or indirectly handles fuzz data, then you should not put 189 the function in a deny instrumentation list and rather live with the 190 instability it comes with. 191 192 4. Recompile the target 193 194 Recompile, fuzz it, be happy :) 195 196 This link explains this process for 197 [Fuzzbench](https://github.com/google/fuzzbench/issues/677). 198