1*387f9dfdSAndroid Build Coastguard WorkerDemonstrations of runqslower, the Linux eBPF/bcc version. 2*387f9dfdSAndroid Build Coastguard Worker 3*387f9dfdSAndroid Build Coastguard Worker 4*387f9dfdSAndroid Build Coastguard Workerrunqslower shows high latency scheduling times between tasks being 5*387f9dfdSAndroid Build Coastguard Workerready to run and them running on CPU after that. For example: 6*387f9dfdSAndroid Build Coastguard Worker 7*387f9dfdSAndroid Build Coastguard Worker# runqslower 8*387f9dfdSAndroid Build Coastguard Worker 9*387f9dfdSAndroid Build Coastguard WorkerNote: Showing TID (thread id) in the report column. The smallest 10*387f9dfdSAndroid Build Coastguard Workerexecution unit becomes a TID when using the --pid flag as 11*387f9dfdSAndroid Build Coastguard Workerin that case the tool reports not only the parent pid but 12*387f9dfdSAndroid Build Coastguard Workerits children threads as well. 13*387f9dfdSAndroid Build Coastguard Worker 14*387f9dfdSAndroid Build Coastguard WorkerTracing run queue latency higher than 10000 us. 15*387f9dfdSAndroid Build Coastguard Worker 16*387f9dfdSAndroid Build Coastguard WorkerTIME COMM TID LAT(us) 17*387f9dfdSAndroid Build Coastguard Worker04:16:32 cc1 12924 12739 18*387f9dfdSAndroid Build Coastguard Worker04:16:32 sh 13640 12118 19*387f9dfdSAndroid Build Coastguard Worker04:16:32 make 13639 12730 20*387f9dfdSAndroid Build Coastguard Worker04:16:32 bash 13655 12047 21*387f9dfdSAndroid Build Coastguard Worker04:16:32 bash 13657 12744 22*387f9dfdSAndroid Build Coastguard Worker04:16:32 bash 13656 12880 23*387f9dfdSAndroid Build Coastguard Worker04:16:32 sh 13660 10846 24*387f9dfdSAndroid Build Coastguard Worker04:16:32 gcc 13663 12681 25*387f9dfdSAndroid Build Coastguard Worker04:16:32 make 13668 10814 26*387f9dfdSAndroid Build Coastguard Worker04:16:32 make 13670 12988 27*387f9dfdSAndroid Build Coastguard Worker04:16:32 gcc 13677 11770 28*387f9dfdSAndroid Build Coastguard Worker04:16:32 gcc 13678 23519 29*387f9dfdSAndroid Build Coastguard Worker04:16:32 as 12999 20541 30*387f9dfdSAndroid Build Coastguard Worker[...] 31*387f9dfdSAndroid Build Coastguard Worker 32*387f9dfdSAndroid Build Coastguard WorkerThis shows various processes waiting for available CPU during a Linux kernel 33*387f9dfdSAndroid Build Coastguard Workerbuild. By default the output contains delays for more than 10ms. 34*387f9dfdSAndroid Build Coastguard Worker 35*387f9dfdSAndroid Build Coastguard WorkerThese delays can be analyzed in depth with "perf sched" tool, see: 36*387f9dfdSAndroid Build Coastguard Worker 37*387f9dfdSAndroid Build Coastguard Worker* http://www.brendangregg.com/blog/2017-03-16/perf-sched.html 38*387f9dfdSAndroid Build Coastguard Worker 39*387f9dfdSAndroid Build Coastguard WorkerUSAGE message: 40*387f9dfdSAndroid Build Coastguard Worker 41*387f9dfdSAndroid Build Coastguard Worker# ./runqslower -h 42*387f9dfdSAndroid Build Coastguard Workerusage: runqslower.py [-h] [-p PID | -t TID] [min_us] 43*387f9dfdSAndroid Build Coastguard Worker 44*387f9dfdSAndroid Build Coastguard WorkerTrace high run queue latency 45*387f9dfdSAndroid Build Coastguard Worker 46*387f9dfdSAndroid Build Coastguard Workerpositional arguments: 47*387f9dfdSAndroid Build Coastguard Worker min_us minimum run queue latency to trace, in us (default 10000) 48*387f9dfdSAndroid Build Coastguard Worker 49*387f9dfdSAndroid Build Coastguard Workeroptional arguments: 50*387f9dfdSAndroid Build Coastguard Worker -h, --help show this help message and exit 51*387f9dfdSAndroid Build Coastguard Worker -p PID, --pid PID trace this PID only 52*387f9dfdSAndroid Build Coastguard Worker -t TID, --tid TID trace this TID only 53*387f9dfdSAndroid Build Coastguard Worker -P, --previous also show previous task name and TID 54*387f9dfdSAndroid Build Coastguard Worker 55*387f9dfdSAndroid Build Coastguard Workerexamples: 56*387f9dfdSAndroid Build Coastguard Worker ./runqslower # trace run queue latency higher than 10000 us (default) 57*387f9dfdSAndroid Build Coastguard Worker ./runqslower 1000 # trace run queue latency higher than 1000 us 58*387f9dfdSAndroid Build Coastguard Worker ./runqslower -p 123 # trace pid 123 59*387f9dfdSAndroid Build Coastguard Worker ./runqslower -t 123 # trace tid 123 (use for threads only) 60*387f9dfdSAndroid Build Coastguard Worker ./runqslower -P # also show previous task comm and TID 61