1Demonstrations of tcppktlat, the Linux BPF CO-RE version. 2 3tcppktlat traces latency between TCP received pkt and picked up by userspace thread system-wide, and prints various details. 4Example output: 5 6# tcppktlat 7PID COMM LADDR LPORT RADDR RPORT MS 84424 etcd 127.0.0.1 2379 127.0.0.1 53202 0.06 94405 kube-apiserver 127.0.0.1 53202 127.0.0.1 2379 0.06 101370211 wakatime-cli 172.22.130.115 59394 143.244.210.202 443 2.75 113894 kubelet 172.18.0.3 51584 172.18.0.4 6443 0.08 12^C 13 14In the process of tracing, when a network soft interrupt reads a TCP packet, it is placed into the corresponding socket's receive queue, and then copied from kernel space to user space by a user-level thread. The latency between receiving the packet and it being retrieved can be used to determine if there are any performance issues in the user-level path for handling received TCP packet. 15 16The -l option can be used to filter on a local port, which is filtered in-kernel. 17 18# tcppktlat -l 7000 19PID COMM LADDR LPORT RADDR RPORT MS 201354840 server 127.0.0.1 7000 127.0.0.1 43932 0.05 211354840 server 127.0.0.1 7000 127.0.0.1 43932 4.88 221354840 server 127.0.0.1 7000 127.0.0.1 43932 6.02 23^C 24 25The output shows server experiences jitter and higher latency in reading data, requiring troubleshooting. 26 27 28We can use minimum latency(us) as a filtering condition. 29 30# tcppktlat 1000 31PID COMM LADDR LPORT RADDR RPORT MS 321354840 server 127.0.0.1 7000 127.0.0.1 35924 130.21 331370211 wakatime-cli 172.22.130.115 59394 143.244.210.202 443 2.75 344405 kube-apiserver :: 6443 :: 55726 1.61 351370227 wakatime-cli 172.22.130.115 50642 143.244.210.202 443 2.80 364405 kube-apiserver 127.0.0.1 53178 127.0.0.1 2379 1.57 371370281 wakatime-cli 172.22.130.115 40908 143.244.210.202 443 3.31 381370315 sshd 127.0.0.1 22 127.0.0.1 42070 6.41 39^C 40 41 42# tcppktlat --help 43Usage: tcppktlat [OPTION...] 44Trace latency between TCP received pkt and picked up by userspace thread. 45 46USAGE: tcppktlat [--help] [-T] [-p PID] [-t TID] [-l LPORT] [-r RPORT] [-v] 47 48EXAMPLES: 49 tcppktlat # Trace all TCP packet picked up latency 50 tcppktlat -T # summarize with timestamps 51 tcppktlat -p # filter for pid 52 tcppktlat -t # filter for tid 53 tcppktlat -l # filter for local port 54 tcppktlat -r # filter for remote port 55 tcppktlat 1000 # filter for latency higher than 1000us 56 57 -l, --lport=LPORT filter for local port 58 -p, --pid=PID Process PID to trace 59 -r, --rport=RPORT filter for remote port 60 -t, --tid=TID Thread TID to trace 61 -T, --timestamp include timestamp on output 62 -v, --verbose Verbose debug output 63 -?, --help Give this help list 64 --usage Give a short usage message 65 -V, --version Print program version 66 67Mandatory or optional arguments to long options are also mandatory or optional 68for any corresponding short options. 69 70Report bugs to https://github.com/iovisor/bcc/tree/master/libbpf-tools. 71