1Some examples for inject 2 3inject guarantees the appropriate erroneous return of the specified injection 4mode (kmalloc,bio,etc) given a call chain and an optional set of predicates. You 5can also optionally print out the generated BPF program for 6modification/debugging purposes. 7 8As a simple example, let's say you wanted to fail all mounts. As of 4.17 we can 9fail syscalls directly, so let's do that: 10 11# ./inject.py kmalloc -v 'SyS_mount()' 12 13The first argument indicates the mode (or what to fail). Appropriate headers are 14specified, if necessary. The verbosity flag prints the generated program. Note 15that some syscalls will be available as 'SyS_xyz' and some will be available as 16'sys_xyz'. This is largely dependent on the number of arguments each syscall 17takes. 18 19Trying to mount various filesystems will fail and report an inability to 20allocate memory, as expected. 21 22Whenever a predicate is missing, an implicit "(true)" is inserted. The example 23above can be explicitly written as: 24 25# ./inject.py kmalloc -v '(true) => SyS_mount()(true)' 26 27The "(true)" without an associated function is a predicate for the error 28injection mechanism of the current mode. In the case of kmalloc, the predicate 29would have access to the arguments of: 30 31 should_failslab(struct kmem_cache *s, gfp_t gfpflags) 32 33Other modes work similarly. 34"bio" has access to the arguments of: 35 36 should_fail_bio(struct bio *bio) 37 38"alloc_page" has access to the arguments of: 39 40 should_fail_alloc_page(gfp_t gfp_mask, unsigned int order) 41 42We also note that it's unnecessary to state the arguments of the function if you 43have no intention to reference them in the associated predicate. 44 45Now let's say we want to be a bit more specific; suppose you want to fail 46kmalloc() from mount_subtree() when called from btrfs_mount(). This will fail 47only btrfs mounts: 48 49# ./inject.py kmalloc -v 'mount_subtree() => btrfs_mount()' 50 51Attempting to mount btrfs filesystem during the execution of this command will 52yield an error, but other filesystems will be fine. 53 54Next, lets say we want to hit one of the BUG_ONs in fs/btrfs. As of 4.16-rc3, 55there is a BUG_ON in btrfs_prepare_close_one_device() at fs/btrfs/volumes.c:1002 56 57To hit this, we can use the following: 58 59# ./inject.py kmalloc -v 'btrfs_alloc_device() => btrfs_close_devices()' 60 61While the script was executing, I mounted and unmounted btrfs, causing a 62segfault on umount(since that satisfied the call path indicated). A look at 63dmesg will confirm that the erroneous return value injected by the script 64tripped the BUG_ON, causing a segfault down the line. 65 66In general, it's worth noting that the required specificity of the call chain is 67dependent on how much granularity you need. The example above might have 68performed as expected without the intermediate btrfs_alloc_device, but might 69have also done something unexpected(an earlier kmalloc could have failed before 70the one we were targeting). 71 72For hot paths, the approach outlined above isn't enough. If a path is traversed 73very often, we can distinguish distinct calls with function arguments. Let's say 74we want to fail the dentry allocation of a file creatively named 'bananas'. We 75can do the following: 76 77# ./inject.py kmalloc -v 'd_alloc_parallel(struct dentry *parent, const struct 78qstr *name)(STRCMP(name->name, 'bananas'))' 79 80While this script is executing, any operation that would cause a dentry 81allocation where the name is 'bananas' fails, as expected. 82 83Here, since we're referencing a function argument in our predicate, we need to 84provide the function signature up to the argument we're using. 85 86To note, STRCMP is a workaround for some rewriter issues. It will take input of 87the form (x->...->z, 'literal'), and generate some equivalent code that the 88verifier is more friendly about. It's not horribly robust, but works for the 89purposes of making string comparisons a bit easier. 90 91Finally, we briefly demonstrate how to inject bio failures. The mechanism is 92identical, so any information from above will apply. 93 94Let's say we want to fail bio requests when the request is to some specific 95sector. An example use case would be to fail superblock writes in btrfs. For 96btrfs, we know that there must be a superblock at 65536 bytes, or sector 128. 97This allows us to run the following: 98 99# ./inject.py bio -v -I 'linux/blkdev.h' '(({struct gendisk *d = bio->bi_disk; 100struct disk_part_tbl *tbl = d->part_tbl; struct hd_struct **parts = (void *)tbl + 101sizeof(struct disk_part_tbl); struct hd_struct **partp = parts + bio->bi_partno; 102struct hd_struct *p = *partp; dev_t disk = p->__dev.devt; disk == 103MKDEV(254,16);}) && bio->bi_iter.bi_sector == 128)' 104 105The predicate in the command above has two parts. The first is a compound 106statement which shortens to "only if the system is btrfs", but is long due 107to rewriter/verifier shenanigans. The major/minor information can be found 108however; I used Python. The second part simply checks the starting 109address of bi_iter. While executing, this script effectively fails superblock 110writes to the superblock at sector 128 without affecting other filesystems. 111 112As an extension to the above, one could easily fail all btrfs superblock writes 113(we only fail the primary) by calculating the sector number of the mirrors and 114amending the predicate accordingly. 115 116Inject also provides a probability option; this allows you to fail the 117path+predicates some percentage of the time. For example, let's say we want to 118fail our mounts half the time: 119 120# ./inject.py kmalloc -v -P 0.01 'SyS_mount()' 121 122USAGE message: 123usage: inject.py [-h] [-I header] [-P probability] [-v] [-c COUNT] 124 {kmalloc,bio,alloc_page} spec 125 126Fail specified kernel functionality when call chain and predicates are met 127 128positional arguments: 129 {kmalloc,bio,alloc_page} 130 indicate which base kernel function to fail 131 spec specify call chain 132 133optional arguments: 134 -h, --help show this help message and exit 135 -I header, --include header 136 additional header files to include in the BPF program 137 -P probability, --probability probability 138 probability that this call chain will fail 139 -v, --verbose print BPF program 140 -c COUNT, --count COUNT 141 Number of fails before bypassing the override 142 143EXAMPLES: 144# ./inject.py kmalloc -v 'SyS_mount()' 145 Fails all calls to syscall mount 146# ./inject.py kmalloc -v '(true) => SyS_mount()(true)' 147 Explicit rewriting of above 148# ./inject.py kmalloc -v 'mount_subtree() => btrfs_mount()' 149 Fails btrfs mounts only 150# ./inject.py kmalloc -v 'd_alloc_parallel(struct dentry *parent, const struct \ 151 qstr *name)(STRCMP(name->name, 'bananas'))' 152 Fails dentry allocations of files named 'bananas' 153# ./inject.py kmalloc -v -P 0.01 'SyS_mount()' 154 Fails calls to syscall mount with 1% probability 155