1*387f9dfdSAndroid Build Coastguard WorkerDemonstrations of filetop, the Linux eBPF/bcc version. 2*387f9dfdSAndroid Build Coastguard Worker 3*387f9dfdSAndroid Build Coastguard Worker 4*387f9dfdSAndroid Build Coastguard Workerfiletop shows reads and writes by file, with process details. For example: 5*387f9dfdSAndroid Build Coastguard Worker 6*387f9dfdSAndroid Build Coastguard Worker# ./filetop -C 7*387f9dfdSAndroid Build Coastguard WorkerTracing... Output every 1 secs. Hit Ctrl-C to end 8*387f9dfdSAndroid Build Coastguard Worker 9*387f9dfdSAndroid Build Coastguard Worker08:00:23 loadavg: 0.91 0.33 0.23 3/286 26635 10*387f9dfdSAndroid Build Coastguard Worker 11*387f9dfdSAndroid Build Coastguard WorkerPID COMM READS WRITES R_Kb W_Kb T FILE 12*387f9dfdSAndroid Build Coastguard Worker26628 ld 161 186 643 152 R built-in.o 13*387f9dfdSAndroid Build Coastguard Worker26634 cc1 1 0 200 0 R autoconf.h 14*387f9dfdSAndroid Build Coastguard Worker26618 cc1 1 0 200 0 R autoconf.h 15*387f9dfdSAndroid Build Coastguard Worker26634 cc1 12 0 192 0 R tracepoint.h 16*387f9dfdSAndroid Build Coastguard Worker26584 cc1 2 0 143 0 R mm.h 17*387f9dfdSAndroid Build Coastguard Worker26634 cc1 2 0 143 0 R mm.h 18*387f9dfdSAndroid Build Coastguard Worker26631 make 34 0 136 0 R auto.conf 19*387f9dfdSAndroid Build Coastguard Worker26634 cc1 1 0 98 0 R fs.h 20*387f9dfdSAndroid Build Coastguard Worker26584 cc1 1 0 98 0 R fs.h 21*387f9dfdSAndroid Build Coastguard Worker26634 cc1 1 0 91 0 R sched.h 22*387f9dfdSAndroid Build Coastguard Worker26634 cc1 1 0 78 0 R printk.c 23*387f9dfdSAndroid Build Coastguard Worker26634 cc1 3 0 73 0 R mmzone.h 24*387f9dfdSAndroid Build Coastguard Worker26628 ld 18 0 72 0 R hibernate.o 25*387f9dfdSAndroid Build Coastguard Worker26628 ld 16 0 64 0 R suspend.o 26*387f9dfdSAndroid Build Coastguard Worker26628 ld 16 0 64 0 R snapshot.o 27*387f9dfdSAndroid Build Coastguard Worker26628 ld 16 0 64 0 R qos.o 28*387f9dfdSAndroid Build Coastguard Worker26628 ld 13 0 52 0 R main.o 29*387f9dfdSAndroid Build Coastguard Worker26628 ld 12 0 52 0 R swap.o 30*387f9dfdSAndroid Build Coastguard Worker[...] 31*387f9dfdSAndroid Build Coastguard Worker 32*387f9dfdSAndroid Build Coastguard WorkerThis shows various files read and written during a Linux kernel build. By 33*387f9dfdSAndroid Build Coastguard Workerdefault the output is sorted by the total read size in Kbytes (R_Kb). Sorting 34*387f9dfdSAndroid Build Coastguard Workerorder can be changed via -s option. This is instrumenting at the VFS interface, 35*387f9dfdSAndroid Build Coastguard Workerso this is reads and writes that may return entirely from the file system cache 36*387f9dfdSAndroid Build Coastguard Worker(page cache). 37*387f9dfdSAndroid Build Coastguard Worker 38*387f9dfdSAndroid Build Coastguard WorkerWhile not printed, the average read and write size can be calculated by 39*387f9dfdSAndroid Build Coastguard Workerdividing R_Kb by READS, and the same for writes. 40*387f9dfdSAndroid Build Coastguard Worker 41*387f9dfdSAndroid Build Coastguard WorkerThe "T" column indicates the type of the file: "R" for regular files, "S" for 42*387f9dfdSAndroid Build Coastguard Workersockets, and "O" for other (including pipes). By default only regular files are 43*387f9dfdSAndroid Build Coastguard Workershown; use the -a option to show all file types. 44*387f9dfdSAndroid Build Coastguard Worker 45*387f9dfdSAndroid Build Coastguard WorkerThis script works by tracing the vfs_read() and vfs_write() functions using 46*387f9dfdSAndroid Build Coastguard Workerkernel dynamic tracing, which instruments explicit read and write calls. If 47*387f9dfdSAndroid Build Coastguard Workerfiles are read or written using another means (eg, via mmap()), then they 48*387f9dfdSAndroid Build Coastguard Workerwill not be visible using this tool. 49*387f9dfdSAndroid Build Coastguard Worker 50*387f9dfdSAndroid Build Coastguard WorkerThis should be useful for file system workload characterization when analyzing 51*387f9dfdSAndroid Build Coastguard Workerthe performance of applications. 52*387f9dfdSAndroid Build Coastguard Worker 53*387f9dfdSAndroid Build Coastguard WorkerNote that tracing VFS level reads and writes can be a frequent activity, and 54*387f9dfdSAndroid Build Coastguard Workerthis tool can begin to cost measurable overhead at high I/O rates. 55*387f9dfdSAndroid Build Coastguard Worker 56*387f9dfdSAndroid Build Coastguard Worker 57*387f9dfdSAndroid Build Coastguard WorkerA -C option will stop clearing the screen, and -r with a number will restrict 58*387f9dfdSAndroid Build Coastguard Workerthe output to that many rows (20 by default). For example, not clearing 59*387f9dfdSAndroid Build Coastguard Workerthe screen and showing the top 5 only: 60*387f9dfdSAndroid Build Coastguard Worker 61*387f9dfdSAndroid Build Coastguard Worker# ./filetop -Cr 5 62*387f9dfdSAndroid Build Coastguard WorkerTracing... Output every 1 secs. Hit Ctrl-C to end 63*387f9dfdSAndroid Build Coastguard Worker 64*387f9dfdSAndroid Build Coastguard Worker08:05:11 loadavg: 0.75 0.35 0.25 3/285 822 65*387f9dfdSAndroid Build Coastguard Worker 66*387f9dfdSAndroid Build Coastguard WorkerPID COMM READS WRITES R_Kb W_Kb T FILE 67*387f9dfdSAndroid Build Coastguard Worker32672 cksum 5006 0 320384 0 R data1 68*387f9dfdSAndroid Build Coastguard Worker809 run 2 0 8 0 R nsswitch.conf 69*387f9dfdSAndroid Build Coastguard Worker811 run 2 0 8 0 R nsswitch.conf 70*387f9dfdSAndroid Build Coastguard Worker804 chown 2 0 8 0 R nsswitch.conf 71*387f9dfdSAndroid Build Coastguard Worker 72*387f9dfdSAndroid Build Coastguard Worker08:05:12 loadavg: 0.75 0.35 0.25 3/285 845 73*387f9dfdSAndroid Build Coastguard Worker 74*387f9dfdSAndroid Build Coastguard WorkerPID COMM READS WRITES R_Kb W_Kb T FILE 75*387f9dfdSAndroid Build Coastguard Worker32672 cksum 4986 0 319104 0 R data1 76*387f9dfdSAndroid Build Coastguard Worker845 chown 2 0 8 0 R nsswitch.conf 77*387f9dfdSAndroid Build Coastguard Worker828 run 2 0 8 0 R nsswitch.conf 78*387f9dfdSAndroid Build Coastguard Worker835 run 2 0 8 0 R nsswitch.conf 79*387f9dfdSAndroid Build Coastguard Worker830 run 2 0 8 0 R nsswitch.conf 80*387f9dfdSAndroid Build Coastguard Worker 81*387f9dfdSAndroid Build Coastguard Worker08:05:13 loadavg: 0.75 0.35 0.25 3/285 868 82*387f9dfdSAndroid Build Coastguard Worker 83*387f9dfdSAndroid Build Coastguard WorkerPID COMM READS WRITES R_Kb W_Kb T FILE 84*387f9dfdSAndroid Build Coastguard Worker32672 cksum 4985 0 319040 0 R data1 85*387f9dfdSAndroid Build Coastguard Worker857 run 2 0 8 0 R nsswitch.conf 86*387f9dfdSAndroid Build Coastguard Worker858 run 2 0 8 0 R nsswitch.conf 87*387f9dfdSAndroid Build Coastguard Worker859 run 2 0 8 0 R nsswitch.conf 88*387f9dfdSAndroid Build Coastguard Worker848 run 2 0 8 0 R nsswitch.conf 89*387f9dfdSAndroid Build Coastguard Worker[...] 90*387f9dfdSAndroid Build Coastguard Worker 91*387f9dfdSAndroid Build Coastguard WorkerThis output shows a cksum command reading data1. 92*387f9dfdSAndroid Build Coastguard Worker 93*387f9dfdSAndroid Build Coastguard Worker 94*387f9dfdSAndroid Build Coastguard WorkerAn optional interval and optional count can also be added to the end of the 95*387f9dfdSAndroid Build Coastguard Workercommand line. For example, for 1 second interval, and 3 summaries in total: 96*387f9dfdSAndroid Build Coastguard Worker 97*387f9dfdSAndroid Build Coastguard Worker# ./filetop -Cr 5 -a 1 3 98*387f9dfdSAndroid Build Coastguard WorkerTracing... Output every 1 secs. Hit Ctrl-C to end 99*387f9dfdSAndroid Build Coastguard Worker 100*387f9dfdSAndroid Build Coastguard Worker08:08:20 loadavg: 0.30 0.42 0.31 3/282 5187 101*387f9dfdSAndroid Build Coastguard Worker 102*387f9dfdSAndroid Build Coastguard WorkerPID COMM READS WRITES R_Kb W_Kb T FILE 103*387f9dfdSAndroid Build Coastguard Worker12421 sshd 14101 0 225616 0 O ptmx 104*387f9dfdSAndroid Build Coastguard Worker12296 sshd 4 0 64 0 O ptmx 105*387f9dfdSAndroid Build Coastguard Worker12421 sshd 3 14104 48 778 S TCP 106*387f9dfdSAndroid Build Coastguard Worker5178 run 2 0 8 0 R nsswitch.conf 107*387f9dfdSAndroid Build Coastguard Worker5165 run 2 0 8 0 R nsswitch.conf 108*387f9dfdSAndroid Build Coastguard Worker 109*387f9dfdSAndroid Build Coastguard Worker08:08:21 loadavg: 0.30 0.42 0.31 5/282 5210 110*387f9dfdSAndroid Build Coastguard Worker 111*387f9dfdSAndroid Build Coastguard WorkerPID COMM READS WRITES R_Kb W_Kb T FILE 112*387f9dfdSAndroid Build Coastguard Worker12421 sshd 9159 0 146544 0 O ptmx 113*387f9dfdSAndroid Build Coastguard Worker12421 sshd 3 9161 48 534 S TCP 114*387f9dfdSAndroid Build Coastguard Worker12296 sshd 1 0 16 0 S TCP 115*387f9dfdSAndroid Build Coastguard Worker5188 run 2 0 8 0 R nsswitch.conf 116*387f9dfdSAndroid Build Coastguard Worker5203 run 2 0 8 0 R nsswitch.conf 117*387f9dfdSAndroid Build Coastguard Worker 118*387f9dfdSAndroid Build Coastguard Worker08:08:22 loadavg: 0.30 0.42 0.31 2/282 5233 119*387f9dfdSAndroid Build Coastguard Worker 120*387f9dfdSAndroid Build Coastguard WorkerPID COMM READS WRITES R_Kb W_Kb T FILE 121*387f9dfdSAndroid Build Coastguard Worker12421 sshd 26166 0 418656 0 O ptmx 122*387f9dfdSAndroid Build Coastguard Worker12421 sshd 4 26171 64 1385 S TCP 123*387f9dfdSAndroid Build Coastguard Worker12296 sshd 1 0 16 0 O ptmx 124*387f9dfdSAndroid Build Coastguard Worker5214 run 2 0 8 0 R nsswitch.conf 125*387f9dfdSAndroid Build Coastguard Worker5227 run 2 0 8 0 R nsswitch.conf 126*387f9dfdSAndroid Build Coastguard WorkerDetaching... 127*387f9dfdSAndroid Build Coastguard Worker 128*387f9dfdSAndroid Build Coastguard WorkerThis example shows the -a option to include all file types. It caught heavy 129*387f9dfdSAndroid Build Coastguard Workersocket I/O from an sshd process, showing up as non-regular file types (the "O" 130*387f9dfdSAndroid Build Coastguard Workerfor other, and "S" for socket, in the type column: "T"). 131*387f9dfdSAndroid Build Coastguard Worker 132*387f9dfdSAndroid Build Coastguard Worker 133*387f9dfdSAndroid Build Coastguard WorkerUSAGE message: 134*387f9dfdSAndroid Build Coastguard Worker 135*387f9dfdSAndroid Build Coastguard Worker# ./filetop -h 136*387f9dfdSAndroid Build Coastguard Workerusage: filetop.py [-h] [-a] [-C] [-r MAXROWS] [-p PID] [interval] [count] 137*387f9dfdSAndroid Build Coastguard Worker 138*387f9dfdSAndroid Build Coastguard WorkerFile reads and writes by process 139*387f9dfdSAndroid Build Coastguard Worker 140*387f9dfdSAndroid Build Coastguard Workerpositional arguments: 141*387f9dfdSAndroid Build Coastguard Worker interval output interval, in seconds 142*387f9dfdSAndroid Build Coastguard Worker count number of outputs 143*387f9dfdSAndroid Build Coastguard Worker 144*387f9dfdSAndroid Build Coastguard Workeroptional arguments: 145*387f9dfdSAndroid Build Coastguard Worker -h, --help show this help message and exit 146*387f9dfdSAndroid Build Coastguard Worker -a, --all-files include non-regular file types (sockets, FIFOs, etc) 147*387f9dfdSAndroid Build Coastguard Worker -C, --noclear don't clear the screen 148*387f9dfdSAndroid Build Coastguard Worker -r MAXROWS, --maxrows MAXROWS 149*387f9dfdSAndroid Build Coastguard Worker maximum rows to print, default 20 150*387f9dfdSAndroid Build Coastguard Worker -s {reads,writes,rbytes,wbytes}, --sort {reads,writes,rbytes,wbytes} 151*387f9dfdSAndroid Build Coastguard Worker sort column, default rbytes 152*387f9dfdSAndroid Build Coastguard Worker -p PID, --pid PID trace this PID only 153*387f9dfdSAndroid Build Coastguard Worker 154*387f9dfdSAndroid Build Coastguard Workerexamples: 155*387f9dfdSAndroid Build Coastguard Worker ./filetop # file I/O top, 1 second refresh 156*387f9dfdSAndroid Build Coastguard Worker ./filetop -C # don't clear the screen 157*387f9dfdSAndroid Build Coastguard Worker ./filetop -p 181 # PID 181 only 158*387f9dfdSAndroid Build Coastguard Worker ./filetop 5 # 5 second summaries 159*387f9dfdSAndroid Build Coastguard Worker ./filetop 5 10 # 5 second summaries, 10 times only 160