1*387f9dfdSAndroid Build Coastguard WorkerDemonstrations of tcptop, the Linux eBPF/bcc version. 2*387f9dfdSAndroid Build Coastguard Worker 3*387f9dfdSAndroid Build Coastguard Worker 4*387f9dfdSAndroid Build Coastguard Workertcptop summarizes throughput by host and port. Eg: 5*387f9dfdSAndroid Build Coastguard Worker 6*387f9dfdSAndroid Build Coastguard Worker# tcptop 7*387f9dfdSAndroid Build Coastguard WorkerTracing... Output every 1 secs. Hit Ctrl-C to end 8*387f9dfdSAndroid Build Coastguard Worker<screen clears> 9*387f9dfdSAndroid Build Coastguard Worker19:46:24 loadavg: 1.86 2.67 2.91 3/362 16681 10*387f9dfdSAndroid Build Coastguard Worker 11*387f9dfdSAndroid Build Coastguard WorkerPID COMM LADDR RADDR RX_KB TX_KB 12*387f9dfdSAndroid Build Coastguard Worker16648 16648 100.66.3.172:22 100.127.69.165:6684 1 0 13*387f9dfdSAndroid Build Coastguard Worker16647 sshd 100.66.3.172:22 100.127.69.165:6684 0 2149 14*387f9dfdSAndroid Build Coastguard Worker14374 sshd 100.66.3.172:22 100.127.69.165:25219 0 0 15*387f9dfdSAndroid Build Coastguard Worker14458 sshd 100.66.3.172:22 100.127.69.165:7165 0 0 16*387f9dfdSAndroid Build Coastguard Worker 17*387f9dfdSAndroid Build Coastguard WorkerPID COMM LADDR6 RADDR6 RX_KB TX_KB 18*387f9dfdSAndroid Build Coastguard Worker16681 sshd fe80::8a3:9dff:fed5:6b19:22 fe80::8a3:9dff:fed5:6b19:16606 1 1 19*387f9dfdSAndroid Build Coastguard Worker16679 ssh fe80::8a3:9dff:fed5:6b19:16606 fe80::8a3:9dff:fed5:6b19:22 1 1 20*387f9dfdSAndroid Build Coastguard Worker16680 sshd fe80::8a3:9dff:fed5:6b19:22 fe80::8a3:9dff:fed5:6b19:16606 0 0 21*387f9dfdSAndroid Build Coastguard Worker 22*387f9dfdSAndroid Build Coastguard WorkerThis example output shows two listings of TCP connections, for IPv4 and IPv6. 23*387f9dfdSAndroid Build Coastguard WorkerIf there is only traffic for one of these, then only one group is shown. 24*387f9dfdSAndroid Build Coastguard Worker 25*387f9dfdSAndroid Build Coastguard WorkerThe output in each listing is sorted by total throughput (send then receive), 26*387f9dfdSAndroid Build Coastguard Workerand when printed it is rounded (floor) to the nearest Kbyte. The example output 27*387f9dfdSAndroid Build Coastguard Workershows PID 16647, sshd, transmitted 2149 Kbytes during the tracing interval. 28*387f9dfdSAndroid Build Coastguard WorkerThe other IPv4 sessions had such low throughput they rounded to zero. 29*387f9dfdSAndroid Build Coastguard Worker 30*387f9dfdSAndroid Build Coastguard WorkerAll TCP sessions, including over loopback, are included. 31*387f9dfdSAndroid Build Coastguard Worker 32*387f9dfdSAndroid Build Coastguard WorkerThe session with the process name (COMM) of 16648 is really a short-lived 33*387f9dfdSAndroid Build Coastguard Workerprocess with PID 16648 where we didn't catch the process name when printing 34*387f9dfdSAndroid Build Coastguard Workerthe output. If this behavior is a serious issue for you, you can modify the 35*387f9dfdSAndroid Build Coastguard Workertool's code to include bpf_get_current_comm() in the key structs, so that it's 36*387f9dfdSAndroid Build Coastguard Workerfetched during the event and will always be seen. I did it this way to start 37*387f9dfdSAndroid Build Coastguard Workerwith, but it was measurably increasing the overhead of this tool, so I switched 38*387f9dfdSAndroid Build Coastguard Workerto the asynchronous model. 39*387f9dfdSAndroid Build Coastguard Worker 40*387f9dfdSAndroid Build Coastguard WorkerThe overhead is relative to TCP event rate (the rate of tcp_sendmsg() and 41*387f9dfdSAndroid Build Coastguard Workertcp_recvmsg() or tcp_cleanup_rbuf()). Due to buffering, this should be lower 42*387f9dfdSAndroid Build Coastguard Workerthan the packet rate. You can measure the rate of these using funccount. 43*387f9dfdSAndroid Build Coastguard WorkerSome sample production servers tested found total rates of 4k to 15k per 44*387f9dfdSAndroid Build Coastguard Workersecond. The CPU overhead at these rates ranged from 0.5% to 2.0% of one CPU. 45*387f9dfdSAndroid Build Coastguard WorkerMaybe your workloads have higher rates and therefore higher overhead, or, 46*387f9dfdSAndroid Build Coastguard Workerlower rates. 47*387f9dfdSAndroid Build Coastguard Worker 48*387f9dfdSAndroid Build Coastguard Worker 49*387f9dfdSAndroid Build Coastguard WorkerI much prefer not clearing the screen, so that historic output is in the 50*387f9dfdSAndroid Build Coastguard Workerscroll-back buffer, and patterns or intermittent issues can be better seen. 51*387f9dfdSAndroid Build Coastguard WorkerYou can do this with -C: 52*387f9dfdSAndroid Build Coastguard Worker 53*387f9dfdSAndroid Build Coastguard Worker# tcptop -C 54*387f9dfdSAndroid Build Coastguard WorkerTracing... Output every 1 secs. Hit Ctrl-C to end 55*387f9dfdSAndroid Build Coastguard Worker 56*387f9dfdSAndroid Build Coastguard Worker20:27:12 loadavg: 0.08 0.02 0.17 2/367 17342 57*387f9dfdSAndroid Build Coastguard Worker 58*387f9dfdSAndroid Build Coastguard WorkerPID COMM LADDR RADDR RX_KB TX_KB 59*387f9dfdSAndroid Build Coastguard Worker17287 17287 100.66.3.172:22 100.127.69.165:57585 3 1 60*387f9dfdSAndroid Build Coastguard Worker17286 sshd 100.66.3.172:22 100.127.69.165:57585 0 1 61*387f9dfdSAndroid Build Coastguard Worker14374 sshd 100.66.3.172:22 100.127.69.165:25219 0 0 62*387f9dfdSAndroid Build Coastguard Worker 63*387f9dfdSAndroid Build Coastguard Worker20:27:13 loadavg: 0.08 0.02 0.17 1/367 17342 64*387f9dfdSAndroid Build Coastguard Worker 65*387f9dfdSAndroid Build Coastguard WorkerPID COMM LADDR RADDR RX_KB TX_KB 66*387f9dfdSAndroid Build Coastguard Worker17286 sshd 100.66.3.172:22 100.127.69.165:57585 1 7761 67*387f9dfdSAndroid Build Coastguard Worker14374 sshd 100.66.3.172:22 100.127.69.165:25219 0 0 68*387f9dfdSAndroid Build Coastguard Worker 69*387f9dfdSAndroid Build Coastguard Worker20:27:14 loadavg: 0.08 0.02 0.17 2/365 17347 70*387f9dfdSAndroid Build Coastguard Worker 71*387f9dfdSAndroid Build Coastguard WorkerPID COMM LADDR RADDR RX_KB TX_KB 72*387f9dfdSAndroid Build Coastguard Worker17286 17286 100.66.3.172:22 100.127.69.165:57585 1 2501 73*387f9dfdSAndroid Build Coastguard Worker14374 sshd 100.66.3.172:22 100.127.69.165:25219 0 0 74*387f9dfdSAndroid Build Coastguard Worker 75*387f9dfdSAndroid Build Coastguard Worker20:27:15 loadavg: 0.07 0.02 0.17 2/367 17403 76*387f9dfdSAndroid Build Coastguard Worker 77*387f9dfdSAndroid Build Coastguard WorkerPID COMM LADDR RADDR RX_KB TX_KB 78*387f9dfdSAndroid Build Coastguard Worker17349 17349 100.66.3.172:22 100.127.69.165:10161 3 1 79*387f9dfdSAndroid Build Coastguard Worker17348 sshd 100.66.3.172:22 100.127.69.165:10161 0 1 80*387f9dfdSAndroid Build Coastguard Worker14374 sshd 100.66.3.172:22 100.127.69.165:25219 0 0 81*387f9dfdSAndroid Build Coastguard Worker 82*387f9dfdSAndroid Build Coastguard Worker20:27:16 loadavg: 0.07 0.02 0.17 1/367 17403 83*387f9dfdSAndroid Build Coastguard Worker 84*387f9dfdSAndroid Build Coastguard WorkerPID COMM LADDR RADDR RX_KB TX_KB 85*387f9dfdSAndroid Build Coastguard Worker17348 sshd 100.66.3.172:22 100.127.69.165:10161 3333 0 86*387f9dfdSAndroid Build Coastguard Worker14374 sshd 100.66.3.172:22 100.127.69.165:25219 0 0 87*387f9dfdSAndroid Build Coastguard Worker 88*387f9dfdSAndroid Build Coastguard Worker20:27:17 loadavg: 0.07 0.02 0.17 2/366 17409 89*387f9dfdSAndroid Build Coastguard Worker 90*387f9dfdSAndroid Build Coastguard WorkerPID COMM LADDR RADDR RX_KB TX_KB 91*387f9dfdSAndroid Build Coastguard Worker17348 17348 100.66.3.172:22 100.127.69.165:10161 6909 2 92*387f9dfdSAndroid Build Coastguard Worker 93*387f9dfdSAndroid Build Coastguard WorkerYou can disable the loadavg summary line with -S if needed. 94*387f9dfdSAndroid Build Coastguard Worker 95*387f9dfdSAndroid Build Coastguard WorkerThe --cgroupmap option filters based on a cgroup set. It is meant to be used 96*387f9dfdSAndroid Build Coastguard Workerwith an externally created map. 97*387f9dfdSAndroid Build Coastguard Worker 98*387f9dfdSAndroid Build Coastguard Worker# tcptop --cgroupmap /sys/fs/bpf/test01 99*387f9dfdSAndroid Build Coastguard Worker 100*387f9dfdSAndroid Build Coastguard WorkerFor more details, see docs/special_filtering.md 101*387f9dfdSAndroid Build Coastguard Worker 102*387f9dfdSAndroid Build Coastguard Worker 103*387f9dfdSAndroid Build Coastguard WorkerUSAGE: 104*387f9dfdSAndroid Build Coastguard Worker 105*387f9dfdSAndroid Build Coastguard Worker# tcptop -h 106*387f9dfdSAndroid Build Coastguard Workerusage: tcptop.py [-h] [-C] [-S] [-p PID] [--cgroupmap CGROUPMAP] 107*387f9dfdSAndroid Build Coastguard Worker [--mntnsmap MNTNSMAP] 108*387f9dfdSAndroid Build Coastguard Worker [interval] [count] [-4 | -6] 109*387f9dfdSAndroid Build Coastguard Worker 110*387f9dfdSAndroid Build Coastguard WorkerSummarize TCP send/recv throughput by host 111*387f9dfdSAndroid Build Coastguard Worker 112*387f9dfdSAndroid Build Coastguard Workerpositional arguments: 113*387f9dfdSAndroid Build Coastguard Worker interval output interval, in seconds (default 1) 114*387f9dfdSAndroid Build Coastguard Worker count number of outputs 115*387f9dfdSAndroid Build Coastguard Worker 116*387f9dfdSAndroid Build Coastguard Workeroptional arguments: 117*387f9dfdSAndroid Build Coastguard Worker -h, --help show this help message and exit 118*387f9dfdSAndroid Build Coastguard Worker -C, --noclear don't clear the screen 119*387f9dfdSAndroid Build Coastguard Worker -S, --nosummary skip system summary line 120*387f9dfdSAndroid Build Coastguard Worker -p PID, --pid PID trace this PID only 121*387f9dfdSAndroid Build Coastguard Worker --cgroupmap CGROUPMAP 122*387f9dfdSAndroid Build Coastguard Worker trace cgroups in this BPF map only 123*387f9dfdSAndroid Build Coastguard Worker -4, --ipv4 trace IPv4 family only 124*387f9dfdSAndroid Build Coastguard Worker -6, --ipv6 trace IPv6 family only 125*387f9dfdSAndroid Build Coastguard Worker 126*387f9dfdSAndroid Build Coastguard Workerexamples: 127*387f9dfdSAndroid Build Coastguard Worker ./tcptop # trace TCP send/recv by host 128*387f9dfdSAndroid Build Coastguard Worker ./tcptop -C # don't clear the screen 129*387f9dfdSAndroid Build Coastguard Worker ./tcptop -p 181 # only trace PID 181 130*387f9dfdSAndroid Build Coastguard Worker ./tcptop --cgroupmap ./mappath # only trace cgroups in this BPF map 131*387f9dfdSAndroid Build Coastguard Worker ./tcptop --mntnsmap mappath # only trace mount namespaces in the map 132*387f9dfdSAndroid Build Coastguard Worker ./tcptop -4 # trace IPv4 family only 133*387f9dfdSAndroid Build Coastguard Worker ./tcptop -6 # trace IPv6 family only 134