1Demonstrations of biolatpcts, the Linux eBPF/bcc version. 2 3 4biolatpcts traces block device I/O (disk I/O), and prints the latency 5percentiles per I/O type. Example: 6 7# ./biolatpcts.py /dev/nvme0n1 8nvme0n1 p1 p5 p10 p16 p25 p50 p75 p84 p90 p95 p99 p100 9read 95us 175us 305us 515us 895us 985us 995us 1.5ms 2.5ms 3.5ms 4.5ms 10ms 10write 5us 5us 5us 15us 25us 135us 765us 855us 885us 895us 965us 1.5ms 11discard 5us 5us 5us 5us 135us 145us 165us 205us 385us 875us 1.5ms 2.5ms 12flush 5us 5us 5us 5us 5us 5us 5us 5us 5us 1.5ms 4.5ms 5.5ms 13[...] 14 15Unless changed with the -i option, the latency percentiles are printed every 3 16seconds. 17 18 19Any number of custom percentiles can be requested with the -p option: 20 21# ./biolatpcts.py /dev/nvme0n1 -p 01,90.0,99.9,99.99,100.0 22 23nvme0n1 01 90.0 99.9 99.99 100.0 24read 5us 4.5ms 16ms 22ms 26ms 25write 15us 255us 365us 515us 2.5ms 26discard - - - - - 27flush 5us 5us 5us 5us 24ms 28[...] 29 30Note that the target percentile strings are preserved as-is to facilitate 31exact key matching when the output is consumed by another program. 32 33 34When the output is consumed by another program, parsing can be tedious. The -j 35option makes biolatpcts output results in json, one line per interval. 36 37# ./tools/biolatpcts.py /dev/nvme0n1 -j 38{"read": {"1": 2.5e-05, "5": 3.5e-05, "10": 4.5e-05, "16": 0.000145, "25": 0.000195, "50": 0.000355, "75": 0.000605, "84": 0.000775, "90": 0.000965, "95": 0.0015, "99": 0.0025, "100": 0.0235}, "write": {"1": 5e-06, "5": 5e-06, "10": 5e-06, "16": 5e-06, "25": 1.5e-05, "50": 2.5e-05, "75": 4.5e-05, "84": 7.5e-05, "90": 0.000195, "95": 0.000665, "99": 0.0015, "100": 0.0035}, "discard": {"1": 0.0, "5": 0.0, "10": 0.0, "16": 0.0, "25": 0.0, "50": 0.0, "75": 0.0, "84": 0.0, "90": 0.0, "95": 0.0, "99": 0.0, "100": 0.0}, "flush": {"1": 0.0, "5": 0.0, "10": 0.0, "16": 0.0, "25": 0.0, "50": 0.0, "75": 0.0, "84": 0.0, "90": 0.0, "95": 0.0, "99": 0.0, "100": 0.0}} 39[...] 40 41 42By default biolatpcts measures the duration each IO was on the device. It can 43be changed using the -w option. 44 45 46USAGE message: 47 48usage: biolatpcts.py [-h] [-i INTERVAL] 49 [-w {from-rq-alloc,after-rq-alloc,on-device}] 50 [-p PCT,...] [-j] [--verbose] 51 DEV 52 53Monitor IO latency distribution of a block device 54 55positional arguments: 56 DEV Target block device (/dev/DEVNAME, DEVNAME or MAJ:MIN) 57 58optional arguments: 59 -h, --help show this help message and exit 60 -i INTERVAL, --interval INTERVAL 61 Report interval (default: 3) 62 -w {from-rq-alloc,after-rq-alloc,on-device}, --which {from-rq-alloc,after-rq-alloc,on-device} 63 Which latency to measure (default: on-device) 64 -p PCT,..., --pcts PCT,... 65 Percentiles to calculate (default: 66 1,5,10,16,25,50,75,84,90,95,99,100) 67 -j, --json Output in json (default: False) 68 --verbose, -v 69