1# Reproducing libFuzzer and AFL crashes 2 3*** note 4**Requirements:** For Windows, you must convert the forward slashes (/) to 5backslashes (\\) in the commands below and use `set` command instead of `export` 6to set the environment variable (step 4). Note that these commands are intended 7to be used with cmd.exe, not PowerShell. Also, you may find [these tips] on how 8to debug an ASAN instrumented binary helpful. 9*** 10 11[TOC] 12 13## Crashes reported as Reproducible 14 15The majority of the bugs reported by ClusterFuzz have **Reproducible** label. 16That means there is a testcase that can be used to reliably reproduce the crash. 17 181. Download the testcase from ClusterFuzz. If you are CCed on an issue filed by 19 ClusterFuzz, a link to it is next to "Reproducer testcase" in the bug 20 description. 21 22 For the rest of this walkthrough, we call the path of this 23 file: `$TESTCASE_PATH` and the fuzz target you want to reproduce a 24 crash on: `$FUZZER_NAME` (provided as "Fuzz Target" in the bug 25 description). 26 272. Generate gn build configuration: 28 29``` 30gn args out/fuzz 31``` 32 33 This will open up an editor. Copy the gn configuration parameters from the 34 values provided in `GN Config` section in the ClusterFuzz testcase report. 35 36 373. Build the fuzzer: 38 39``` 40autoninja -C out/fuzz $FUZZER_NAME 41``` 42 434. Set the `*SAN_OPTIONS` environment variable as provided in the 44 `Crash Stacktrace` section in the testcase report. 45 Here is an example value of `ASAN_OPTIONS` that is similar to its value on 46 ClusterFuzz: 47 48``` 49export ASAN_OPTIONS=redzone=256:print_summary=1:handle_sigill=1:allocator_release_to_os_interval_ms=500:print_suppressions=0:strict_memcmp=1:allow_user_segv_handler=0:use_sigaltstack=1:handle_sigfpe=1:handle_sigbus=1:detect_stack_use_after_return=0:alloc_dealloc_mismatch=0:detect_leaks=0:print_scariness=1:allocator_may_return_null=1:handle_abort=1:check_malloc_usable_size=0:detect_container_overflow=0:quarantine_size_mb=256:detect_odr_violation=0:symbolize=1:handle_segv=1:fast_unwind_on_fatal=0 50``` 51 525. Run the fuzz target: 53 54``` 55out/fuzz/$FUZZER_NAME -runs=100 $TESTCASE_PATH 56``` 57 58[File a bug] if you run into any issues. 59 60## Symbolizing stack traces 61 62Stack traces from ASAN builds are not symbolized by default. However, you 63can symbolize them by piping the output into: 64 65``` 66src/tools/valgrind/asan/asan_symbolize.py 67``` 68 69## Crashes reported as Unreproducible 70 71ClusterFuzz generally does not report issues that it cannot reliably reproduce, 72unless the following condition is met. If a certain crash is occurring often 73enough, such a crash might be reported with **Unreproducible** label and an 74explicit clarification that there is no convenient way to reproduce it. There 75are two ways to work with such crashes. 76 771. Try a speculative fix based on the stacktrace. Once the fix is landed, wait a 78 couple days and then check Crash Statistics section on the ClusterFuzz 79 testcase report page. If the fix works out, you will see that the crash is 80 not happening anymore. If the crash does not occur again for a little while, 81 ClusterFuzz will automatically close the issue as Verified. 82 832. (libFuzzer only) Try to reproduce the whole fuzzing session. This workflow is 84 very similar to the one described above for the **Reproducible** crashes. The 85 only differences are: 86 87 * On step 1, instead of downloading a single testcase, you need to download 88 corpus backup. This can be done using the following command: 89``` 90gsutil cp gs://clusterfuzz-libfuzzer-backup/corpus/libfuzzer/$FUZZER_NAME/latest.zip . 91``` 92 93 * Alternatively, you can navigate to the following URL in your browser and 94 download the `latest.zip` file: 95``` 96https://pantheon.corp.google.com/storage/browser/clusterfuzz-libfuzzer-backup/corpus/libfuzzer/$FUZZER_NAME 97``` 98 99 * Create an empty directory and unpack the corpus into it. 100 * Follow steps 2-4 in the **Reproducible** section above. 101 * On step 5, use the following command: 102 103``` 104out/fuzz/$FUZZER_NAME -timeout=25 -rss_limit_mb=2048 -print_final_stats=1 $CORPUS_DIRECTORY_FROM_THE_PREVIOUS_STEP 105``` 106 107 * Wait and hope that the fuzzer will crash. 108 109Waiting for a crash to occur may take some time (up to 1hr), but if it happens, 110you will be able to test the fix locally and/or somehow debug the issue. 111 112## Minimizing a crash input (optional) 113 114ClusterFuzz does crash input minimization automatically, and a typical crash 115report has two testcases available for downloading: 116 117* An original testcase that has triggered the crash; 118* A minimized testcase that is smaller than the original but triggers the same 119 crash. 120 121If you would like to further minimize a testcase, run the fuzz target with the 122two additional arguments: 123 124* `-minimize_crash=1` 125* `-exact_artifact_path=<output_filename_for_minimized_testcase>` 126 127The full command would be: 128 129``` 130out/fuzz/$FUZZER_NAME -minimize_crash=1 -exact_artifact_path=<minimized_testcase_path> $TESTCASE_PATH 131``` 132 133This might be useful for large testcases that make it hard to identify a root 134cause of a crash. You can leave the minimization running locally for a while 135(e.g. overnight) for better results. 136 137 138[File a bug]: https://bugs.chromium.org/p/chromium/issues/entry?components=Tools%3EStability%3ElibFuzzer&comment=What%20problem%20are%20you%20seeing 139[these tips]: https://github.com/google/sanitizers/wiki/AddressSanitizerWindowsPort#debugging 140