1Demonstrations of tcprtt, the Linux eBPF/bcc version. 2 3 4This program traces TCP RTT(round-trip time) to analyze the quality of 5network, then help us to distinguish the network latency trouble is from 6user process or physical network. 7 8For example, wrk show the http request latency distribution: 9# wrk -d 30 -c 10 --latency http://192.168.122.100/index.html 10Running 30s test @ http://192.168.122.100/index.html 11 2 threads and 10 connections 12 Thread Stats Avg Stdev Max +/- Stdev 13 Latency 86.75ms 153.76ms 1.54s 90.85% 14 Req/Sec 160.91 76.07 424.00 67.06% 15 Latency Distribution 16 50% 14.55ms 17 75% 119.21ms 18 90% 230.22ms 19 99% 726.90ms 20 9523 requests in 30.02s, 69.62MB read 21 Socket errors: connect 0, read 0, write 0, timeout 1 22 23During wrk testing, run tcprtt: 24# ./tcprtt -i 1 -d 10 -m 25Tracing TCP RTT... Hit Ctrl-C to end. 26 msecs : count distribution 27 0 -> 1 : 4 | | 28 2 -> 3 : 0 | | 29 4 -> 7 : 1055 |****************************************| 30 8 -> 15 : 26 | | 31 16 -> 31 : 0 | | 32 32 -> 63 : 0 | | 33 64 -> 127 : 18 | | 34 128 -> 255 : 14 | | 35 256 -> 511 : 14 | | 36 512 -> 1023 : 12 | | 37 38The wrk output shows that the latency of web service is not stable, and tcprtt 39also shows unstable TCP RTT. So in this situation, we need to make sure the 40quality of network is good or not firstly. 41 42 43Use filter for address and(or) port. Ex, only collect local address 192.168.122.200 44and remote address 192.168.122.100 and remote port 80. 45# ./tcprtt -i 1 -d 10 -m -a 192.168.122.200 -A 192.168.122.100 -P 80 46 47 48Tracing at server side, show each clients with its own histogram. 49For example, run tcprtt on a storage node to show initiators' rtt histogram: 50# ./tcprtt -i 1 --lport 3260 --byraddr -e 51Tracing TCP RTT... Hit Ctrl-C to end. 52 53 54Remote Addres = 10.194.87.206 [AVG 170] 55 usecs : count distribution 56 0 -> 1 : 0 | | 57 2 -> 3 : 0 | | 58 4 -> 7 : 0 | | 59 8 -> 15 : 0 | | 60 16 -> 31 : 0 | | 61 32 -> 63 : 31 | | 62 64 -> 127 : 5150 |******************* | 63 128 -> 255 : 10327 |****************************************| 64 256 -> 511 : 1014 |*** | 65 512 -> 1023 : 10 | | 66 1024 -> 2047 : 7 | | 67 2048 -> 4095 : 14 | | 68 4096 -> 8191 : 10 | | 69 70Remote Addres = 10.194.87.197 [AVG 4293] 71 usecs : count distribution 72 0 -> 1 : 0 | | 73 2 -> 3 : 0 | | 74 4 -> 7 : 0 | | 75 8 -> 15 : 0 | | 76 16 -> 31 : 0 | | 77 32 -> 63 : 0 | | 78 64 -> 127 : 0 | | 79 128 -> 255 : 0 | | 80 256 -> 511 : 0 | | 81 512 -> 1023 : 0 | | 82 1024 -> 2047 : 3 |******** | 83 2048 -> 4095 : 12 |********************************** | 84 4096 -> 8191 : 14 |****************************************| 85 86Remote Addres = 10.194.88.148 [AVG 6215] 87 usecs : count distribution 88 0 -> 1 : 0 | | 89 2 -> 3 : 0 | | 90 4 -> 7 : 0 | | 91 8 -> 15 : 0 | | 92 16 -> 31 : 0 | | 93 32 -> 63 : 0 | | 94 64 -> 127 : 0 | | 95 128 -> 255 : 0 | | 96 256 -> 511 : 0 | | 97 512 -> 1023 : 0 | | 98 1024 -> 2047 : 0 | | 99 2048 -> 4095 : 0 | | 100 4096 -> 8191 : 2 |****************************************| 101 102Remote Addres = 10.194.87.90 [AVG 2188] 103 usecs : count distribution 104 0 -> 1 : 0 | | 105 2 -> 3 : 0 | | 106 4 -> 7 : 0 | | 107 8 -> 15 : 0 | | 108 16 -> 31 : 0 | | 109 32 -> 63 : 0 | | 110 64 -> 127 : 0 | | 111 128 -> 255 : 0 | | 112 256 -> 511 : 15 |********* | 113 512 -> 1023 : 30 |****************** | 114 1024 -> 2047 : 50 |****************************** | 115 2048 -> 4095 : 65 |****************************************| 116 4096 -> 8191 : 22 |************* | 117 118.... 119 120 121Use -e(--extension) to show extension RTT: 122# ./tcprtt -i 1 -e 123 124All Addresses = ******* [AVG 324] 125 usecs : count distribution 126 0 -> 1 : 0 | | 127 2 -> 3 : 0 | | 128 4 -> 7 : 0 | | 129 8 -> 15 : 0 | | 130 16 -> 31 : 0 | | 131 32 -> 63 : 0 | | 132 64 -> 127 : 5360 |******** | 133 128 -> 255 : 23834 |****************************************| 134 256 -> 511 : 11276 |****************** | 135 512 -> 1023 : 700 |* | 136 1024 -> 2047 : 434 | | 137 2048 -> 4095 : 356 | | 138 4096 -> 8191 : 328 | | 139 8192 -> 16383 : 91 | | 140 141 142Full USAGE: 143 144# ./tcprtt -h 145usage: tcprtt [-h] [-i INTERVAL] [-d DURATION] [-T] [-m] [-p LPORT] 146 [-P RPORT] [-a LADDR] [-A RADDR] [-b] [-B] [-e] [-D] 147 [-4 | -6] 148 149Summarize TCP RTT as a histogram 150 151optional arguments: 152 -h, --help show this help message and exit 153 -i INTERVAL, --interval INTERVAL 154 summary interval, seconds 155 -d DURATION, --duration DURATION 156 total duration of trace, seconds 157 -T, --timestamp include timestamp on output 158 -m, --milliseconds millisecond histogram 159 -p LPORT, --lport LPORT 160 filter for local port 161 -P RPORT, --rport RPORT 162 filter for remote port 163 -a LADDR, --laddr LADDR 164 filter for local address 165 -A RADDR, --raddr RADDR 166 filter for remote address 167 -b, --byladdr show sockets histogram by local address 168 -B, --byraddr show sockets histogram by remote address 169 -e, --extension show extension summary(average) 170 -D, --debug print BPF program before starting (for debugging 171 purposes) 172 -4, --ipv4 trace IPv4 family only 173 -6, --ipv6 trace IPv6 family only 174 175examples: 176 ./tcprtt # summarize TCP RTT 177 ./tcprtt -i 1 -d 10 # print 1 second summaries, 10 times 178 ./tcprtt -m -T # summarize in millisecond, and timestamps 179 ./tcprtt -p # filter for local port 180 ./tcprtt -P # filter for remote port 181 ./tcprtt -a # filter for local address 182 ./tcprtt -A # filter for remote address 183 ./tcprtt -b # show sockets histogram by local address 184 ./tcprtt -B # show sockets histogram by remote address 185 ./tcprtt -D # show debug bpf text 186 ./tcprtt -e # show extension summary(average) 187 ./tcprtt -4 # trace IPv4 family only 188 ./tcprtt -6 # trace IPv6 family only 189