1*387f9dfdSAndroid Build Coastguard WorkerDemonstrations of biotop, the Linux eBPF/bcc version. 2*387f9dfdSAndroid Build Coastguard Worker 3*387f9dfdSAndroid Build Coastguard Worker 4*387f9dfdSAndroid Build Coastguard WorkerShort for block device I/O top, biotop summarizes which processes are 5*387f9dfdSAndroid Build Coastguard Workerperforming disk I/O. It's top for disks. Sample output: 6*387f9dfdSAndroid Build Coastguard Worker 7*387f9dfdSAndroid Build Coastguard Worker# ./biotop 8*387f9dfdSAndroid Build Coastguard WorkerTracing... Output every 1 secs. Hit Ctrl-C to end 9*387f9dfdSAndroid Build Coastguard Worker 10*387f9dfdSAndroid Build Coastguard Worker08:04:11 loadavg: 1.48 0.87 0.45 1/287 14547 11*387f9dfdSAndroid Build Coastguard Worker 12*387f9dfdSAndroid Build Coastguard WorkerPID COMM D MAJ MIN DISK I/O Kbytes AVGms 13*387f9dfdSAndroid Build Coastguard Worker14501 cksum R 202 1 xvda1 361 28832 3.39 14*387f9dfdSAndroid Build Coastguard Worker6961 dd R 202 1 xvda1 1628 13024 0.59 15*387f9dfdSAndroid Build Coastguard Worker13855 dd R 202 1 xvda1 1627 13016 0.59 16*387f9dfdSAndroid Build Coastguard Worker326 jbd2/xvda1-8 W 202 1 xvda1 3 168 3.00 17*387f9dfdSAndroid Build Coastguard Worker1880 supervise W 202 1 xvda1 2 8 6.71 18*387f9dfdSAndroid Build Coastguard Worker1873 supervise W 202 1 xvda1 2 8 2.51 19*387f9dfdSAndroid Build Coastguard Worker1871 supervise W 202 1 xvda1 2 8 1.57 20*387f9dfdSAndroid Build Coastguard Worker1876 supervise W 202 1 xvda1 2 8 1.22 21*387f9dfdSAndroid Build Coastguard Worker1892 supervise W 202 1 xvda1 2 8 0.62 22*387f9dfdSAndroid Build Coastguard Worker1878 supervise W 202 1 xvda1 2 8 0.78 23*387f9dfdSAndroid Build Coastguard Worker1886 supervise W 202 1 xvda1 2 8 1.30 24*387f9dfdSAndroid Build Coastguard Worker1894 supervise W 202 1 xvda1 2 8 3.46 25*387f9dfdSAndroid Build Coastguard Worker1869 supervise W 202 1 xvda1 2 8 0.73 26*387f9dfdSAndroid Build Coastguard Worker1888 supervise W 202 1 xvda1 2 8 1.48 27*387f9dfdSAndroid Build Coastguard Worker 28*387f9dfdSAndroid Build Coastguard WorkerBy default the screen refreshes every 1 second, and shows the top 20 disk 29*387f9dfdSAndroid Build Coastguard Workerconsumers, sorted on total Kbytes. The first line printed is the header, 30*387f9dfdSAndroid Build Coastguard Workerwhich has the time and then the contents of /proc/loadavg. 31*387f9dfdSAndroid Build Coastguard Worker 32*387f9dfdSAndroid Build Coastguard WorkerFor the interval summarized by the output above, the "cksum" command performed 33*387f9dfdSAndroid Build Coastguard Worker361 disk reads to the "xvda1" device, for a total of 28832 Kbytes, with an 34*387f9dfdSAndroid Build Coastguard Workeraverage I/O time of 3.39 ms. Two "dd" processes were also reading from the 35*387f9dfdSAndroid Build Coastguard Workersame disk, which a higher I/O rate and lower latency. While the average I/O 36*387f9dfdSAndroid Build Coastguard Workersize is not printed, it can be determined by dividing the Kbytes column by 37*387f9dfdSAndroid Build Coastguard Workerthe I/O column. 38*387f9dfdSAndroid Build Coastguard Worker 39*387f9dfdSAndroid Build Coastguard WorkerThe columns through to Kbytes show the workload applied. The final column, 40*387f9dfdSAndroid Build Coastguard WorkerAVGms, shows resulting performance. Other bcc tools can be used to get more 41*387f9dfdSAndroid Build Coastguard Workerdetails when needed: biolatency and biosnoop. 42*387f9dfdSAndroid Build Coastguard Worker 43*387f9dfdSAndroid Build Coastguard WorkerMany years ago I created the original "iotop", and later regretted not calling 44*387f9dfdSAndroid Build Coastguard Workerit diskiotop or blockiotop, as "io" alone is ambiguous. This time it is biotop. 45*387f9dfdSAndroid Build Coastguard Worker 46*387f9dfdSAndroid Build Coastguard Worker 47*387f9dfdSAndroid Build Coastguard WorkerThe -C option can be used to prevent the screen from clearing (my preference). 48*387f9dfdSAndroid Build Coastguard WorkerHere's using it with a 5 second interval: 49*387f9dfdSAndroid Build Coastguard Worker 50*387f9dfdSAndroid Build Coastguard Worker# ./biotop -C 5 51*387f9dfdSAndroid Build Coastguard WorkerTracing... Output every 5 secs. Hit Ctrl-C to end 52*387f9dfdSAndroid Build Coastguard Worker 53*387f9dfdSAndroid Build Coastguard Worker08:09:44 loadavg: 0.42 0.44 0.39 2/282 22115 54*387f9dfdSAndroid Build Coastguard Worker 55*387f9dfdSAndroid Build Coastguard WorkerPID COMM D MAJ MIN DISK I/O Kbytes AVGms 56*387f9dfdSAndroid Build Coastguard Worker22069 dd R 202 1 xvda1 5993 47976 0.33 57*387f9dfdSAndroid Build Coastguard Worker326 jbd2/xvda1-8 W 202 1 xvda1 3 168 2.67 58*387f9dfdSAndroid Build Coastguard Worker1866 svscan R 202 1 xvda1 33 132 1.24 59*387f9dfdSAndroid Build Coastguard Worker1880 supervise W 202 1 xvda1 10 40 0.56 60*387f9dfdSAndroid Build Coastguard Worker1873 supervise W 202 1 xvda1 10 40 0.79 61*387f9dfdSAndroid Build Coastguard Worker1871 supervise W 202 1 xvda1 10 40 0.78 62*387f9dfdSAndroid Build Coastguard Worker1876 supervise W 202 1 xvda1 10 40 0.68 63*387f9dfdSAndroid Build Coastguard Worker1892 supervise W 202 1 xvda1 10 40 0.71 64*387f9dfdSAndroid Build Coastguard Worker1878 supervise W 202 1 xvda1 10 40 0.65 65*387f9dfdSAndroid Build Coastguard Worker1886 supervise W 202 1 xvda1 10 40 0.78 66*387f9dfdSAndroid Build Coastguard Worker1894 supervise W 202 1 xvda1 10 40 0.80 67*387f9dfdSAndroid Build Coastguard Worker1869 supervise W 202 1 xvda1 10 40 0.91 68*387f9dfdSAndroid Build Coastguard Worker1888 supervise W 202 1 xvda1 10 40 0.63 69*387f9dfdSAndroid Build Coastguard Worker22069 bash R 202 1 xvda1 1 16 19.94 70*387f9dfdSAndroid Build Coastguard Worker9251 kworker/u16:2 W 202 16 xvdb 2 8 0.13 71*387f9dfdSAndroid Build Coastguard Worker 72*387f9dfdSAndroid Build Coastguard Worker08:09:49 loadavg: 0.47 0.44 0.39 1/282 22231 73*387f9dfdSAndroid Build Coastguard Worker 74*387f9dfdSAndroid Build Coastguard WorkerPID COMM D MAJ MIN DISK I/O Kbytes AVGms 75*387f9dfdSAndroid Build Coastguard Worker22069 dd R 202 1 xvda1 13450 107600 0.35 76*387f9dfdSAndroid Build Coastguard Worker22199 cksum R 202 1 xvda1 941 45548 4.63 77*387f9dfdSAndroid Build Coastguard Worker326 jbd2/xvda1-8 W 202 1 xvda1 3 168 2.93 78*387f9dfdSAndroid Build Coastguard Worker24467 kworker/0:2 W 202 16 xvdb 1 64 0.28 79*387f9dfdSAndroid Build Coastguard Worker1880 supervise W 202 1 xvda1 10 40 0.81 80*387f9dfdSAndroid Build Coastguard Worker1873 supervise W 202 1 xvda1 10 40 0.81 81*387f9dfdSAndroid Build Coastguard Worker1871 supervise W 202 1 xvda1 10 40 1.03 82*387f9dfdSAndroid Build Coastguard Worker1876 supervise W 202 1 xvda1 10 40 0.76 83*387f9dfdSAndroid Build Coastguard Worker1892 supervise W 202 1 xvda1 10 40 0.74 84*387f9dfdSAndroid Build Coastguard Worker1878 supervise W 202 1 xvda1 10 40 0.94 85*387f9dfdSAndroid Build Coastguard Worker1886 supervise W 202 1 xvda1 10 40 0.76 86*387f9dfdSAndroid Build Coastguard Worker1894 supervise W 202 1 xvda1 10 40 0.69 87*387f9dfdSAndroid Build Coastguard Worker1869 supervise W 202 1 xvda1 10 40 0.72 88*387f9dfdSAndroid Build Coastguard Worker1888 supervise W 202 1 xvda1 10 40 1.70 89*387f9dfdSAndroid Build Coastguard Worker22199 bash R 202 1 xvda1 2 20 0.35 90*387f9dfdSAndroid Build Coastguard Worker482 xfsaild/md0 W 202 16 xvdb 5 13 0.27 91*387f9dfdSAndroid Build Coastguard Worker482 xfsaild/md0 W 202 32 xvdc 2 8 0.33 92*387f9dfdSAndroid Build Coastguard Worker31331 pickup R 202 1 xvda1 1 4 0.31 93*387f9dfdSAndroid Build Coastguard Worker 94*387f9dfdSAndroid Build Coastguard Worker08:09:54 loadavg: 0.51 0.45 0.39 2/282 22346 95*387f9dfdSAndroid Build Coastguard Worker 96*387f9dfdSAndroid Build Coastguard WorkerPID COMM D MAJ MIN DISK I/O Kbytes AVGms 97*387f9dfdSAndroid Build Coastguard Worker22069 dd R 202 1 xvda1 14689 117512 0.32 98*387f9dfdSAndroid Build Coastguard Worker326 jbd2/xvda1-8 W 202 1 xvda1 3 168 2.33 99*387f9dfdSAndroid Build Coastguard Worker1880 supervise W 202 1 xvda1 10 40 0.65 100*387f9dfdSAndroid Build Coastguard Worker1873 supervise W 202 1 xvda1 10 40 1.08 101*387f9dfdSAndroid Build Coastguard Worker1871 supervise W 202 1 xvda1 10 40 0.66 102*387f9dfdSAndroid Build Coastguard Worker1876 supervise W 202 1 xvda1 10 40 0.79 103*387f9dfdSAndroid Build Coastguard Worker1892 supervise W 202 1 xvda1 10 40 0.67 104*387f9dfdSAndroid Build Coastguard Worker1878 supervise W 202 1 xvda1 10 40 0.66 105*387f9dfdSAndroid Build Coastguard Worker1886 supervise W 202 1 xvda1 10 40 1.02 106*387f9dfdSAndroid Build Coastguard Worker1894 supervise W 202 1 xvda1 10 40 0.88 107*387f9dfdSAndroid Build Coastguard Worker1869 supervise W 202 1 xvda1 10 40 0.89 108*387f9dfdSAndroid Build Coastguard Worker1888 supervise W 202 1 xvda1 10 40 1.25 109*387f9dfdSAndroid Build Coastguard Worker 110*387f9dfdSAndroid Build Coastguard Worker08:09:59 loadavg: 0.55 0.46 0.40 2/282 22461 111*387f9dfdSAndroid Build Coastguard Worker 112*387f9dfdSAndroid Build Coastguard WorkerPID COMM D MAJ MIN DISK I/O Kbytes AVGms 113*387f9dfdSAndroid Build Coastguard Worker22069 dd R 202 1 xvda1 14442 115536 0.33 114*387f9dfdSAndroid Build Coastguard Worker326 jbd2/xvda1-8 W 202 1 xvda1 3 168 3.46 115*387f9dfdSAndroid Build Coastguard Worker1880 supervise W 202 1 xvda1 10 40 0.87 116*387f9dfdSAndroid Build Coastguard Worker1873 supervise W 202 1 xvda1 10 40 0.87 117*387f9dfdSAndroid Build Coastguard Worker1871 supervise W 202 1 xvda1 10 40 0.78 118*387f9dfdSAndroid Build Coastguard Worker1876 supervise W 202 1 xvda1 10 40 0.86 119*387f9dfdSAndroid Build Coastguard Worker1892 supervise W 202 1 xvda1 10 40 0.89 120*387f9dfdSAndroid Build Coastguard Worker1878 supervise W 202 1 xvda1 10 40 0.87 121*387f9dfdSAndroid Build Coastguard Worker1886 supervise W 202 1 xvda1 10 40 0.86 122*387f9dfdSAndroid Build Coastguard Worker1894 supervise W 202 1 xvda1 10 40 1.06 123*387f9dfdSAndroid Build Coastguard Worker1869 supervise W 202 1 xvda1 10 40 1.12 124*387f9dfdSAndroid Build Coastguard Worker1888 supervise W 202 1 xvda1 10 40 0.98 125*387f9dfdSAndroid Build Coastguard Worker 126*387f9dfdSAndroid Build Coastguard Worker08:10:04 loadavg: 0.59 0.47 0.40 3/282 22576 127*387f9dfdSAndroid Build Coastguard Worker 128*387f9dfdSAndroid Build Coastguard WorkerPID COMM D MAJ MIN DISK I/O Kbytes AVGms 129*387f9dfdSAndroid Build Coastguard Worker22069 dd R 202 1 xvda1 14179 113432 0.34 130*387f9dfdSAndroid Build Coastguard Worker326 jbd2/xvda1-8 W 202 1 xvda1 3 168 2.39 131*387f9dfdSAndroid Build Coastguard Worker1880 supervise W 202 1 xvda1 10 40 0.81 132*387f9dfdSAndroid Build Coastguard Worker1873 supervise W 202 1 xvda1 10 40 1.02 133*387f9dfdSAndroid Build Coastguard Worker1871 supervise W 202 1 xvda1 10 40 1.15 134*387f9dfdSAndroid Build Coastguard Worker1876 supervise W 202 1 xvda1 10 40 1.10 135*387f9dfdSAndroid Build Coastguard Worker1892 supervise W 202 1 xvda1 10 40 0.77 136*387f9dfdSAndroid Build Coastguard Worker1878 supervise W 202 1 xvda1 10 40 0.72 137*387f9dfdSAndroid Build Coastguard Worker1886 supervise W 202 1 xvda1 10 40 0.81 138*387f9dfdSAndroid Build Coastguard Worker1894 supervise W 202 1 xvda1 10 40 0.86 139*387f9dfdSAndroid Build Coastguard Worker1869 supervise W 202 1 xvda1 10 40 0.83 140*387f9dfdSAndroid Build Coastguard Worker1888 supervise W 202 1 xvda1 10 40 0.79 141*387f9dfdSAndroid Build Coastguard Worker24467 kworker/0:2 R 202 32 xvdc 3 12 0.26 142*387f9dfdSAndroid Build Coastguard Worker1056 cron R 202 1 xvda1 2 8 0.30 143*387f9dfdSAndroid Build Coastguard Worker24467 kworker/0:2 R 202 16 xvdb 1 4 0.23 144*387f9dfdSAndroid Build Coastguard Worker 145*387f9dfdSAndroid Build Coastguard Worker08:10:09 loadavg: 0.54 0.46 0.40 2/281 22668 146*387f9dfdSAndroid Build Coastguard Worker 147*387f9dfdSAndroid Build Coastguard WorkerPID COMM D MAJ MIN DISK I/O Kbytes AVGms 148*387f9dfdSAndroid Build Coastguard Worker22069 dd R 202 1 xvda1 250 2000 0.34 149*387f9dfdSAndroid Build Coastguard Worker326 jbd2/xvda1-8 W 202 1 xvda1 3 168 2.40 150*387f9dfdSAndroid Build Coastguard Worker1880 supervise W 202 1 xvda1 8 32 0.93 151*387f9dfdSAndroid Build Coastguard Worker1873 supervise W 202 1 xvda1 8 32 0.76 152*387f9dfdSAndroid Build Coastguard Worker1871 supervise W 202 1 xvda1 8 32 0.60 153*387f9dfdSAndroid Build Coastguard Worker1876 supervise W 202 1 xvda1 8 32 0.61 154*387f9dfdSAndroid Build Coastguard Worker1892 supervise W 202 1 xvda1 8 32 0.68 155*387f9dfdSAndroid Build Coastguard Worker1878 supervise W 202 1 xvda1 8 32 0.90 156*387f9dfdSAndroid Build Coastguard Worker1886 supervise W 202 1 xvda1 8 32 0.57 157*387f9dfdSAndroid Build Coastguard Worker1894 supervise W 202 1 xvda1 8 32 0.97 158*387f9dfdSAndroid Build Coastguard Worker1869 supervise W 202 1 xvda1 8 32 0.69 159*387f9dfdSAndroid Build Coastguard Worker1888 supervise W 202 1 xvda1 8 32 0.67 160*387f9dfdSAndroid Build Coastguard Worker 161*387f9dfdSAndroid Build Coastguard WorkerThis shows another "dd" command reading from xvda1. On this system, various 162*387f9dfdSAndroid Build Coastguard Worker"supervise" processes do 8 disk writes per second, every second (they are 163*387f9dfdSAndroid Build Coastguard Workercreating and updating "status" files). 164*387f9dfdSAndroid Build Coastguard Worker 165*387f9dfdSAndroid Build Coastguard Worker 166*387f9dfdSAndroid Build Coastguard WorkerUSAGE message: 167*387f9dfdSAndroid Build Coastguard Worker 168*387f9dfdSAndroid Build Coastguard Worker# ./biotop.py -h 169*387f9dfdSAndroid Build Coastguard Workerusage: biotop.py [-h] [-C] [-r MAXROWS] [interval] [count] 170*387f9dfdSAndroid Build Coastguard Worker 171*387f9dfdSAndroid Build Coastguard WorkerBlock device (disk) I/O by process 172*387f9dfdSAndroid Build Coastguard Worker 173*387f9dfdSAndroid Build Coastguard Workerpositional arguments: 174*387f9dfdSAndroid Build Coastguard Worker interval output interval, in seconds 175*387f9dfdSAndroid Build Coastguard Worker count number of outputs 176*387f9dfdSAndroid Build Coastguard Worker 177*387f9dfdSAndroid Build Coastguard Workeroptional arguments: 178*387f9dfdSAndroid Build Coastguard Worker -h, --help show this help message and exit 179*387f9dfdSAndroid Build Coastguard Worker -C, --noclear don't clear the screen 180*387f9dfdSAndroid Build Coastguard Worker -r MAXROWS, --maxrows MAXROWS 181*387f9dfdSAndroid Build Coastguard Worker maximum rows to print, default 20 182*387f9dfdSAndroid Build Coastguard Worker 183*387f9dfdSAndroid Build Coastguard Workerexamples: 184*387f9dfdSAndroid Build Coastguard Worker ./biotop # block device I/O top, 1 second refresh 185*387f9dfdSAndroid Build Coastguard Worker ./biotop -C # don't clear the screen 186*387f9dfdSAndroid Build Coastguard Worker ./biotop 5 # 5 second summaries 187*387f9dfdSAndroid Build Coastguard Worker ./biotop 5 10 # 5 second summaries, 10 times only 188