1# Getting started with fuzzing in Chromium 2 3This document walks through how to get started adding fuzz tests to Chromium. 4 5It guides you how to use our latest fuzzing technology, called [FuzzTest]. This 6replaces earlier technology called [libfuzzer]. See the section at the end 7for reasons why you might sometimes still want to use libfuzzer. 8 9[TOC] 10 11## What to fuzz 12 13You should fuzz any function which takes input from any 14untrusted source, such as the internet. If the code parses, decodes, or 15otherwise manipulates that input, it definitely should be fuzzed! 16 17To decide how best to fuzz it, you should decide which of these two situations 18best matches your input: 19 20* *Binary data*: the input is a single buffer of contiguous bytes, for example 21 an image or some binary format which your code decodes. 22* *Function arguments*: the input is multiple different chunks of data, for 23 example the arguments to a function. 24 25In the latter case, go ahead and read this guide - it will show you how to use 26our latest fuzzing technology, FuzzTest. 27 28If however your input more closely matches the former description - just a 29single binary blob of data - then instead use our older fuzzing technology 30[libfuzzer] - click that link for a separate getting started guide. (libfuzzer 31will work a little better in these cases because if the fuzzer finds a problem, 32the test case will exactly match the binary format.) 33 34## How to fuzz 35 361. Find your existing unit test target. Create a new similar target 37 alongside. (In the future, you'll be able to add them right into your 38 unit test code directly.) 392. Add a gn target definition a lot like a normal unit test, but with 40 `fuzztests = [ list-of-fuzztests ]`. See below for details. Create a `.cc` file. 413. In the unit tests code, `#include "third_party/fuzztest/src/fuzztest/fuzztest.h"` 424. Add a `FUZZ_TEST` macro, which might be as simple as `FUZZ_TEST(MyApiTest, ExistingFunctionWhichTakesUntrustedInput)` 43 (though you may wish to structure things differently, see below) 445. Run the unit tests and ensure they pass. 456. Land the CL. 46 47That's it! 48 49This fuzzer will be built automatically, using various [sanitizers], and run 50on our distributed fuzzing infrastructure [ClusterFuzz]. If it finds bugs, 51they'll be reported back to you. 52 53More detail in all the following sections. 54 55## Creating a new `FUZZ_TEST` target 56 57``` 58import("//testing/test.gni") 59 60test("hypothetical_fuzztests") { 61 sources = [ "hypothetical_fuzztests.cc" ] 62 63 fuzztests = ["MyApiTest.MyApiCanSuccessfullyParseAnyString"] 64 65 deps = [ 66 ":hypothetical_component", 67 "//third_party/fuzztest:fuzztest_gtest_main", 68 ] 69} 70``` 71 72You may also need to add `third_party/fuzztest` to your DEPS file. 73 74## Adding `FUZZ_TEST` support to a target 75 76*** note 77**Note:** Currently, you must create a **new** unit test target. 78While the FuzzTest framework supports mixed unit and fuzz tests, 79we don't yet support this option in Chromium. 80*** 81 82In the near future we'll support adding `FUZZ_TEST`s alongside existing 83unit tests, even in the same .cc file. 84 85``` 86if (is_linux) { 87 test("existing_unit_tests") { 88 sources = [ "existing_unit_tests.cc" ] # add FUZZ_TESTs here 89 90 fuzztests = ["MyApiTest.ApiWorksAlways"] 91 # Add this! 92 93 deps = [ 94 ":existing_component", 95 # Other stuff 96 ] 97 } 98} 99``` 100 101This will: 102* add a dependency on the appropriate fuzztest libraries; 103* cause the target to be built on all our [fuzzer builders] 104* construct metadata so that [ClusterFuzz] knows how to run the resulting 105 binary. 106 107This relies on something, somewhere, calling `base::LaunchUnitTests` within 108your executable to initialize FuzzTest. This should be the case already. 109 110(If you have other code targets, such as `source_set`s, contributing to your 111unit test target they may need to explicitly depend upon `//third_party/fuzztest` 112too.) 113 114*** note 115**Note:** Again, this is not yet supported! 116*** 117 118## Adding `FUZZ_TEST`s in the code 119 120First, `#include "third_party/fuzztest/src/fuzztest/fuzztest.h"`. 121 122Then, it's normal to create a function named after the thing you're trying to 123prove, with assertions to prove it. 124 125For instance, 126 127``` 128void MyApiCanSuccessfullyParseAnyString(std::string input) { 129 bool success = MyApi(input); 130 EXPECT_TRUE(success); 131} 132``` 133 134Then, declare the `FUZZ_TEST` macro: 135 136``` 137FUZZ_TEST(MyApiTest, MyApiCanSuccessfullyParseAnyString); 138``` 139 140Our fuzzing infrastructure will generate all possible strings and prove it works. 141Obviously, that takes infinite time, so instead our fuzzing infrastructure will 142carefully craft strings to explore more and more branches within `MyApi`, 143mutating the input according to code coverage, so there's a good chance bugs 144will be found quickly. 145 146Fuzzing should always be alongside traditional unit testing - never rely on it 147to find all the bugs! It should be a backstop to prevent unexpected security 148flaws sneaking past your regular testing. 149 150In more complex cases, you'll need to tell FuzzTest about the expected domains 151of valid input. For example: 152 153``` 154void MyApiAlwaysSucceedsOnPositiveIntegers(int i) { 155 bool success = MyApi(i); 156 EXPECT_TRUE(success); 157} 158FUZZ_TEST(MyApiTest, MyApiAlwaysSucceedsOnPositiveIntegers) 159 .WithDomains(/*i:*/fuzztest::Positive<int>()); 160``` 161 162See the [FuzzTest reference] for all your options here. 163 164## Running this locally 165 166Simply build and run your unit tests as normal. `FUZZ_TEST`s are supported only 167on some platforms. If you're on such a platform, you'll see your fuzz test 168run for one second: 169 170``` 171[==========] Running 1 test from 1 test suite. 172[----------] Global test environment set-up. 173[----------] 1 test from ScaleFuzz 174[ RUN ] ApiTest.MyApiCanSuccessfullyParseAnyString 175[ OK ] ApiTest.MyApiCanSuccessfullyParseAnyString (1000 ms) 176[----------] 1 test from ScaleFuzz (1000 ms total) 177 178[----------] Global test environment tear-down 179[==========] 1 test from 1 test suite ran. (1000 ms total) 180[ PASSED ] 1 test. 181``` 182 183On other platforms, the test will be ignored. 184 185If you want to try actually fuzzing with FuzzTest, modify your gn arguments to 186contain: 187 188``` 189enable_fuzztest_fuzz=true 190is_component_build=false 191``` 192 193You can then run your unit test with the extra command line argument `--fuzz=`, 194optionally specifying a test name. You'll see lots of output as it explores your 195code: 196 197``` 198[*] Corpus size: 1 | Edges covered: 73 | Fuzzing time: 1.60482ms | Total runs: 1.00e+00 | Runs/secs: 623 | Max stack usage: 0 199[*] Corpus size: 2 | Edges covered: 103 | Fuzzing time: 1.844ms | Total runs: 2.00e+00 | Runs/secs: 1084 | Max stack usage: 0 200[*] Corpus size: 3 | Edges covered: 111 | Fuzzing time: 2.747931ms | Total runs: 3.00e+00 | Runs/secs: 1091 | Max stack usage: 0 201[*] Corpus size: 4 | Edges covered: 135 | Fuzzing time: 2.92305ms | Total runs: 4.00e+00 | Runs/secs: 1368 | Max stack usage: 0 202[*] Corpus size: 5 | Edges covered: 173 | Fuzzing time: 3.35237ms | Total runs: 5.00e+00 | Runs/secs: 1491 | Max stack usage: 0 203[*] Corpus size: 6 | Edges covered: 178 | Fuzzing time: 4.15666ms | Total runs: 6.00e+00 | Runs/secs: 1443 | Max stack usage: 0 204``` 205 206("Edges covered") is how many different code blocks have been explored (that is, 207sections between branches). Over time, you'll see it explore more and more until 208it runs out of new edges to explore. 209 210## Landing the CL 211 212Nothing special is required here! 213 214After a day or two, we should see [ClusterFuzz] starting to run your new fuzzer, 215and it should be visible on [ClusterFuzz Fuzzer Stats]. Look for fuzzers starting 216with `centipede_` and your test target's name. 217 218*** note 219**Note:** This is all very new, and ClusterFuzz isn't reliably spotting these 220new fuzztests yet. We're working on it! 221*** 222 223Thanks very much for doing your part in making Chromium more secure! 224 225## Unusual cases 226 227There are some situations where FuzzTests may not work. For example: 228 229* You need to run on platforms not currently supported by FuzzTest 230* You need more structured input 231* You need to mutate the input in a more precise way 232* Your fuzzer input is a single binary blob 233 234In these cases, you may be best off creating a standalone fuzzer using our 235older fuzzing technology, [libfuzzer]. There are further options beyond 236that, e.g. uploading "black box" fuzzers to ClusterFuzz, or even running 237fuzzers outside of ClusterFuzz which then upload results to ClusterFuzz 238for triage and diagnosis. To explore any of those options, please discuss 239with the fuzzing team (email [email protected] if you're outside Google). 240 241 242[FuzzTest]: https://github.com/google/fuzztest#how-do-i-use-it 243[libfuzzer]: getting_started_with_libfuzzer.md 244[`test` template]: https://source.chromium.org/chromium/chromium/src/+/main:testing/test.gni?q=test.gni 245[fuzzer builders]: https://ci.chromium.org/p/chromium/g/chromium.fuzz/console 246[ClusterFuzz]: https://clusterfuzz.com/ 247[FuzzTest reference]: https://github.com/google/fuzztest#how-do-i-use-it 248[ClusterFuzz Fuzzer Stats]: https://clusterfuzz.com/fuzzer-stats/by-fuzzer/fuzzer/libFuzzer/job/libfuzzer_chrome_asan 249