1*387f9dfdSAndroid Build Coastguard WorkerDemonstrations of biosnoop, the Linux eBPF/bcc version. 2*387f9dfdSAndroid Build Coastguard Worker 3*387f9dfdSAndroid Build Coastguard Worker 4*387f9dfdSAndroid Build Coastguard Workerbiosnoop traces block device I/O (disk I/O), and prints a line of output 5*387f9dfdSAndroid Build Coastguard Workerper I/O. Example: 6*387f9dfdSAndroid Build Coastguard Worker 7*387f9dfdSAndroid Build Coastguard Worker# ./biosnoop 8*387f9dfdSAndroid Build Coastguard WorkerTIME(s) COMM PID DISK T SECTOR BYTES LAT(ms) 9*387f9dfdSAndroid Build Coastguard Worker0.000004 supervise 1950 xvda1 W 13092560 4096 0.74 10*387f9dfdSAndroid Build Coastguard Worker0.000178 supervise 1950 xvda1 W 13092432 4096 0.61 11*387f9dfdSAndroid Build Coastguard Worker0.001469 supervise 1956 xvda1 W 13092440 4096 1.24 12*387f9dfdSAndroid Build Coastguard Worker0.001588 supervise 1956 xvda1 W 13115128 4096 1.09 13*387f9dfdSAndroid Build Coastguard Worker1.022346 supervise 1950 xvda1 W 13115272 4096 0.98 14*387f9dfdSAndroid Build Coastguard Worker1.022568 supervise 1950 xvda1 W 13188496 4096 0.93 15*387f9dfdSAndroid Build Coastguard Worker1.023534 supervise 1956 xvda1 W 13188520 4096 0.79 16*387f9dfdSAndroid Build Coastguard Worker1.023585 supervise 1956 xvda1 W 13189512 4096 0.60 17*387f9dfdSAndroid Build Coastguard Worker2.003920 xfsaild/md0 456 xvdc W 62901512 8192 0.23 18*387f9dfdSAndroid Build Coastguard Worker2.003931 xfsaild/md0 456 xvdb W 62901513 512 0.25 19*387f9dfdSAndroid Build Coastguard Worker2.004034 xfsaild/md0 456 xvdb W 62901520 8192 0.35 20*387f9dfdSAndroid Build Coastguard Worker2.004042 xfsaild/md0 456 xvdb W 63542016 4096 0.36 21*387f9dfdSAndroid Build Coastguard Worker2.004204 kworker/0:3 26040 xvdb W 41950344 65536 0.34 22*387f9dfdSAndroid Build Coastguard Worker2.044352 supervise 1950 xvda1 W 13192672 4096 0.65 23*387f9dfdSAndroid Build Coastguard Worker2.044574 supervise 1950 xvda1 W 13189072 4096 0.58 24*387f9dfdSAndroid Build Coastguard Worker 25*387f9dfdSAndroid Build Coastguard WorkerThis includes the PID and comm (process name) that were on-CPU at the time of 26*387f9dfdSAndroid Build Coastguard Workerissue (which usually means the process responsible). 27*387f9dfdSAndroid Build Coastguard Worker 28*387f9dfdSAndroid Build Coastguard WorkerThe latency of the disk I/O, measured from the issue to the device to its 29*387f9dfdSAndroid Build Coastguard Workercompletion, is included as the last column. 30*387f9dfdSAndroid Build Coastguard Worker 31*387f9dfdSAndroid Build Coastguard WorkerThis example output is from what should be an idle system, however, the 32*387f9dfdSAndroid Build Coastguard Workerfollowing is visible in iostat: 33*387f9dfdSAndroid Build Coastguard Worker 34*387f9dfdSAndroid Build Coastguard Worker$ iostat -x 1 35*387f9dfdSAndroid Build Coastguard Worker[...] 36*387f9dfdSAndroid Build Coastguard Workeravg-cpu: %user %nice %system %iowait %steal %idle 37*387f9dfdSAndroid Build Coastguard Worker 0.12 0.00 0.12 0.00 0.00 99.75 38*387f9dfdSAndroid Build Coastguard Worker 39*387f9dfdSAndroid Build Coastguard WorkerDevice: rrqm/s wrqm/s r/s w/s rkB/s wkB/s await svctm %util 40*387f9dfdSAndroid Build Coastguard Workerxvda 0.00 0.00 0.00 4.00 0.00 16.00 0.00 0.00 0.00 41*387f9dfdSAndroid Build Coastguard Workerxvdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 42*387f9dfdSAndroid Build Coastguard Workerxvdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 43*387f9dfdSAndroid Build Coastguard Workermd0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 44*387f9dfdSAndroid Build Coastguard Worker 45*387f9dfdSAndroid Build Coastguard WorkerThere are 4 write IOPS. 46*387f9dfdSAndroid Build Coastguard Worker 47*387f9dfdSAndroid Build Coastguard WorkerThe output of biosnoop identifies the reason: multiple supervise processes are 48*387f9dfdSAndroid Build Coastguard Workerissuing writes to the xvda1 disk. I can now drill down on supervise using other 49*387f9dfdSAndroid Build Coastguard Workertools to understand its file system workload. 50*387f9dfdSAndroid Build Coastguard Worker 51*387f9dfdSAndroid Build Coastguard Worker 52*387f9dfdSAndroid Build Coastguard WorkerThe -Q option includes a column to show the time spent queued in the OS: 53*387f9dfdSAndroid Build Coastguard Worker 54*387f9dfdSAndroid Build Coastguard Worker# biosnoop.py -Q 55*387f9dfdSAndroid Build Coastguard WorkerTIME(s) COMM PID DISK T SECTOR BYTES QUE(ms) LAT(ms) 56*387f9dfdSAndroid Build Coastguard Worker0.000000 kworker/u72:1 13379 nvme1n1 W 1142400 4096 0.01 0.55 57*387f9dfdSAndroid Build Coastguard Worker0.000771 sync 22177 nvme1n1 W 41963894 3072 0.11 0.47 58*387f9dfdSAndroid Build Coastguard Worker5.332998 xfsaild/nvme1n 1061 nvme1n1 W 545728 16384 0.01 0.61 59*387f9dfdSAndroid Build Coastguard Worker5.333044 xfsaild/nvme1n 1061 nvme1n1 W 2349728 16384 0.02 0.64 60*387f9dfdSAndroid Build Coastguard Worker5.333065 xfsaild/nvme1n 1061 nvme1n1 W 20971521 512 0.02 0.65 61*387f9dfdSAndroid Build Coastguard Worker5.333067 xfsaild/nvme1n 1061 nvme1n1 W 20971528 8192 0.00 0.65 62*387f9dfdSAndroid Build Coastguard Worker[...] 63*387f9dfdSAndroid Build Coastguard Worker 64*387f9dfdSAndroid Build Coastguard Worker 65*387f9dfdSAndroid Build Coastguard WorkerUSAGE message: 66*387f9dfdSAndroid Build Coastguard Worker 67*387f9dfdSAndroid Build Coastguard Workerusage: biosnoop.py [-h] [-Q] [-d DISK] [-P] 68*387f9dfdSAndroid Build Coastguard Worker 69*387f9dfdSAndroid Build Coastguard WorkerTrace block I/O 70*387f9dfdSAndroid Build Coastguard Worker 71*387f9dfdSAndroid Build Coastguard Workeroptional arguments: 72*387f9dfdSAndroid Build Coastguard Worker -h, --help show this help message and exit 73*387f9dfdSAndroid Build Coastguard Worker -Q, --queue include OS queued time 74*387f9dfdSAndroid Build Coastguard Worker -d DISK, --disk DISK trace this disk only 75*387f9dfdSAndroid Build Coastguard Worker -P, --pattern display block I/O pattern (sequential or random) 76*387f9dfdSAndroid Build Coastguard Worker 77*387f9dfdSAndroid Build Coastguard Workerexamples: 78*387f9dfdSAndroid Build Coastguard Worker ./biosnoop # trace all block I/O 79*387f9dfdSAndroid Build Coastguard Worker ./biosnoop -Q # include OS queued time 80*387f9dfdSAndroid Build Coastguard Worker ./biosnoop -d sdc # trace sdc only 81*387f9dfdSAndroid Build Coastguard Worker ./biosnoop -P # display block I/O pattern 82