1*387f9dfdSAndroid Build Coastguard WorkerDemonstrations of biolatpcts, the Linux eBPF/bcc version. 2*387f9dfdSAndroid Build Coastguard Worker 3*387f9dfdSAndroid Build Coastguard Worker 4*387f9dfdSAndroid Build Coastguard Workerbiolatpcts traces block device I/O (disk I/O), and prints the latency 5*387f9dfdSAndroid Build Coastguard Workerpercentiles per I/O type. Example: 6*387f9dfdSAndroid Build Coastguard Worker 7*387f9dfdSAndroid Build Coastguard Worker# ./biolatpcts.py /dev/nvme0n1 8*387f9dfdSAndroid Build Coastguard Workernvme0n1 p1 p5 p10 p16 p25 p50 p75 p84 p90 p95 p99 p100 9*387f9dfdSAndroid Build Coastguard Workerread 95us 175us 305us 515us 895us 985us 995us 1.5ms 2.5ms 3.5ms 4.5ms 10ms 10*387f9dfdSAndroid Build Coastguard Workerwrite 5us 5us 5us 15us 25us 135us 765us 855us 885us 895us 965us 1.5ms 11*387f9dfdSAndroid Build Coastguard Workerdiscard 5us 5us 5us 5us 135us 145us 165us 205us 385us 875us 1.5ms 2.5ms 12*387f9dfdSAndroid Build Coastguard Workerflush 5us 5us 5us 5us 5us 5us 5us 5us 5us 1.5ms 4.5ms 5.5ms 13*387f9dfdSAndroid Build Coastguard Worker[...] 14*387f9dfdSAndroid Build Coastguard Worker 15*387f9dfdSAndroid Build Coastguard WorkerUnless changed with the -i option, the latency percentiles are printed every 3 16*387f9dfdSAndroid Build Coastguard Workerseconds. 17*387f9dfdSAndroid Build Coastguard Worker 18*387f9dfdSAndroid Build Coastguard Worker 19*387f9dfdSAndroid Build Coastguard WorkerAny number of custom percentiles can be requested with the -p option: 20*387f9dfdSAndroid Build Coastguard Worker 21*387f9dfdSAndroid Build Coastguard Worker# ./biolatpcts.py /dev/nvme0n1 -p 01,90.0,99.9,99.99,100.0 22*387f9dfdSAndroid Build Coastguard Worker 23*387f9dfdSAndroid Build Coastguard Workernvme0n1 01 90.0 99.9 99.99 100.0 24*387f9dfdSAndroid Build Coastguard Workerread 5us 4.5ms 16ms 22ms 26ms 25*387f9dfdSAndroid Build Coastguard Workerwrite 15us 255us 365us 515us 2.5ms 26*387f9dfdSAndroid Build Coastguard Workerdiscard - - - - - 27*387f9dfdSAndroid Build Coastguard Workerflush 5us 5us 5us 5us 24ms 28*387f9dfdSAndroid Build Coastguard Worker[...] 29*387f9dfdSAndroid Build Coastguard Worker 30*387f9dfdSAndroid Build Coastguard WorkerNote that the target percentile strings are preserved as-is to facilitate 31*387f9dfdSAndroid Build Coastguard Workerexact key matching when the output is consumed by another program. 32*387f9dfdSAndroid Build Coastguard Worker 33*387f9dfdSAndroid Build Coastguard Worker 34*387f9dfdSAndroid Build Coastguard WorkerWhen the output is consumed by another program, parsing can be tedious. The -j 35*387f9dfdSAndroid Build Coastguard Workeroption makes biolatpcts output results in json, one line per interval. 36*387f9dfdSAndroid Build Coastguard Worker 37*387f9dfdSAndroid Build Coastguard Worker# ./tools/biolatpcts.py /dev/nvme0n1 -j 38*387f9dfdSAndroid Build Coastguard Worker{"read": {"1": 2.5e-05, "5": 3.5e-05, "10": 4.5e-05, "16": 0.000145, "25": 0.000195, "50": 0.000355, "75": 0.000605, "84": 0.000775, "90": 0.000965, "95": 0.0015, "99": 0.0025, "100": 0.0235}, "write": {"1": 5e-06, "5": 5e-06, "10": 5e-06, "16": 5e-06, "25": 1.5e-05, "50": 2.5e-05, "75": 4.5e-05, "84": 7.5e-05, "90": 0.000195, "95": 0.000665, "99": 0.0015, "100": 0.0035}, "discard": {"1": 0.0, "5": 0.0, "10": 0.0, "16": 0.0, "25": 0.0, "50": 0.0, "75": 0.0, "84": 0.0, "90": 0.0, "95": 0.0, "99": 0.0, "100": 0.0}, "flush": {"1": 0.0, "5": 0.0, "10": 0.0, "16": 0.0, "25": 0.0, "50": 0.0, "75": 0.0, "84": 0.0, "90": 0.0, "95": 0.0, "99": 0.0, "100": 0.0}} 39*387f9dfdSAndroid Build Coastguard Worker[...] 40*387f9dfdSAndroid Build Coastguard Worker 41*387f9dfdSAndroid Build Coastguard Worker 42*387f9dfdSAndroid Build Coastguard WorkerBy default biolatpcts measures the duration each IO was on the device. It can 43*387f9dfdSAndroid Build Coastguard Workerbe changed using the -w option. 44*387f9dfdSAndroid Build Coastguard Worker 45*387f9dfdSAndroid Build Coastguard Worker 46*387f9dfdSAndroid Build Coastguard WorkerUSAGE message: 47*387f9dfdSAndroid Build Coastguard Worker 48*387f9dfdSAndroid Build Coastguard Workerusage: biolatpcts.py [-h] [-i INTERVAL] 49*387f9dfdSAndroid Build Coastguard Worker [-w {from-rq-alloc,after-rq-alloc,on-device}] 50*387f9dfdSAndroid Build Coastguard Worker [-p PCT,...] [-j] [--verbose] 51*387f9dfdSAndroid Build Coastguard Worker DEV 52*387f9dfdSAndroid Build Coastguard Worker 53*387f9dfdSAndroid Build Coastguard WorkerMonitor IO latency distribution of a block device 54*387f9dfdSAndroid Build Coastguard Worker 55*387f9dfdSAndroid Build Coastguard Workerpositional arguments: 56*387f9dfdSAndroid Build Coastguard Worker DEV Target block device (/dev/DEVNAME, DEVNAME or MAJ:MIN) 57*387f9dfdSAndroid Build Coastguard Worker 58*387f9dfdSAndroid Build Coastguard Workeroptional arguments: 59*387f9dfdSAndroid Build Coastguard Worker -h, --help show this help message and exit 60*387f9dfdSAndroid Build Coastguard Worker -i INTERVAL, --interval INTERVAL 61*387f9dfdSAndroid Build Coastguard Worker Report interval (default: 3) 62*387f9dfdSAndroid Build Coastguard Worker -w {from-rq-alloc,after-rq-alloc,on-device}, --which {from-rq-alloc,after-rq-alloc,on-device} 63*387f9dfdSAndroid Build Coastguard Worker Which latency to measure (default: on-device) 64*387f9dfdSAndroid Build Coastguard Worker -p PCT,..., --pcts PCT,... 65*387f9dfdSAndroid Build Coastguard Worker Percentiles to calculate (default: 66*387f9dfdSAndroid Build Coastguard Worker 1,5,10,16,25,50,75,84,90,95,99,100) 67*387f9dfdSAndroid Build Coastguard Worker -j, --json Output in json (default: False) 68*387f9dfdSAndroid Build Coastguard Worker --verbose, -v 69