1*387f9dfdSAndroid Build Coastguard WorkerDemonstrations of stacksnoop, the Linux eBPF/bcc version. 2*387f9dfdSAndroid Build Coastguard Worker 3*387f9dfdSAndroid Build Coastguard Worker 4*387f9dfdSAndroid Build Coastguard WorkerThis program traces the given kernel function and prints the kernel stack trace 5*387f9dfdSAndroid Build Coastguard Workerfor every call. This tool is useful for studying low frequency kernel functions, 6*387f9dfdSAndroid Build Coastguard Workerto see how they were invoked. For example, tracing the submit_bio() call: 7*387f9dfdSAndroid Build Coastguard Worker 8*387f9dfdSAndroid Build Coastguard Worker# ./stacksnoop submit_bio 9*387f9dfdSAndroid Build Coastguard WorkerTIME(s) SYSCALL 10*387f9dfdSAndroid Build Coastguard Worker3592.838736000 submit_bio 11*387f9dfdSAndroid Build Coastguard Worker submit_bio 12*387f9dfdSAndroid Build Coastguard Worker submit_bh 13*387f9dfdSAndroid Build Coastguard Worker jbd2_journal_commit_transaction 14*387f9dfdSAndroid Build Coastguard Worker kjournald2 15*387f9dfdSAndroid Build Coastguard Worker kthread 16*387f9dfdSAndroid Build Coastguard Worker ret_from_fork 17*387f9dfdSAndroid Build Coastguard Worker 18*387f9dfdSAndroid Build Coastguard WorkerThis shows that submit_bio() was called by submit_bh(), which was called 19*387f9dfdSAndroid Build Coastguard Workerby jbd2_journal_commit_transaction(), and so on. 20*387f9dfdSAndroid Build Coastguard Worker 21*387f9dfdSAndroid Build Coastguard WorkerFor high frequency functions, see stackcount, which summarizes in-kernel for 22*387f9dfdSAndroid Build Coastguard Workerefficiency. If you don't know if your function is low or high frequency, try 23*387f9dfdSAndroid Build Coastguard Workerfunccount. 24*387f9dfdSAndroid Build Coastguard Worker 25*387f9dfdSAndroid Build Coastguard Worker 26*387f9dfdSAndroid Build Coastguard WorkerThe -v option includes more fields, including the on-CPU process (COMM and PID): 27*387f9dfdSAndroid Build Coastguard Worker 28*387f9dfdSAndroid Build Coastguard Worker# ./stacksnoop -v submit_bio 29*387f9dfdSAndroid Build Coastguard WorkerTIME(s) COMM PID CPU SYSCALL 30*387f9dfdSAndroid Build Coastguard Worker3734.855027000 jbd2/dm-0-8 313 0 submit_bio 31*387f9dfdSAndroid Build Coastguard Worker submit_bio 32*387f9dfdSAndroid Build Coastguard Worker submit_bh 33*387f9dfdSAndroid Build Coastguard Worker jbd2_journal_commit_transaction 34*387f9dfdSAndroid Build Coastguard Worker kjournald2 35*387f9dfdSAndroid Build Coastguard Worker kthread 36*387f9dfdSAndroid Build Coastguard Worker ret_from_fork 37*387f9dfdSAndroid Build Coastguard Worker 38*387f9dfdSAndroid Build Coastguard WorkerThis identifies the application issuing the sync syscall: the jbd2 process 39*387f9dfdSAndroid Build Coastguard Worker(COMM column). 40*387f9dfdSAndroid Build Coastguard Worker 41*387f9dfdSAndroid Build Coastguard Worker 42*387f9dfdSAndroid Build Coastguard WorkerHere's another example, showing the path to second_overflow() and on-CPU 43*387f9dfdSAndroid Build Coastguard Workerprocess: 44*387f9dfdSAndroid Build Coastguard Worker 45*387f9dfdSAndroid Build Coastguard Worker# ./stacksnoop -v second_overflow 46*387f9dfdSAndroid Build Coastguard WorkerTIME(s) COMM PID CPU SYSCALL 47*387f9dfdSAndroid Build Coastguard Worker3837.526433000 <idle> 0 1 second_overflow 48*387f9dfdSAndroid Build Coastguard Worker second_overflow 49*387f9dfdSAndroid Build Coastguard Worker tick_do_update_jiffies64 50*387f9dfdSAndroid Build Coastguard Worker tick_irq_enter 51*387f9dfdSAndroid Build Coastguard Worker irq_enter 52*387f9dfdSAndroid Build Coastguard Worker smp_apic_timer_interrupt 53*387f9dfdSAndroid Build Coastguard Worker apic_timer_interrupt 54*387f9dfdSAndroid Build Coastguard Worker default_idle 55*387f9dfdSAndroid Build Coastguard Worker arch_cpu_idle 56*387f9dfdSAndroid Build Coastguard Worker default_idle_call 57*387f9dfdSAndroid Build Coastguard Worker cpu_startup_entry 58*387f9dfdSAndroid Build Coastguard Worker start_secondary 59*387f9dfdSAndroid Build Coastguard Worker 60*387f9dfdSAndroid Build Coastguard Worker3838.526953000 <idle> 0 1 second_overflow 61*387f9dfdSAndroid Build Coastguard Worker second_overflow 62*387f9dfdSAndroid Build Coastguard Worker tick_do_update_jiffies64 63*387f9dfdSAndroid Build Coastguard Worker tick_irq_enter 64*387f9dfdSAndroid Build Coastguard Worker irq_enter 65*387f9dfdSAndroid Build Coastguard Worker smp_apic_timer_interrupt 66*387f9dfdSAndroid Build Coastguard Worker apic_timer_interrupt 67*387f9dfdSAndroid Build Coastguard Worker default_idle 68*387f9dfdSAndroid Build Coastguard Worker arch_cpu_idle 69*387f9dfdSAndroid Build Coastguard Worker default_idle_call 70*387f9dfdSAndroid Build Coastguard Worker cpu_startup_entry 71*387f9dfdSAndroid Build Coastguard Worker start_secondary 72*387f9dfdSAndroid Build Coastguard Worker 73*387f9dfdSAndroid Build Coastguard WorkerThis fires every second (see TIME(s)), and is from tick_do_update_jiffies64(). 74*387f9dfdSAndroid Build Coastguard Worker 75*387f9dfdSAndroid Build Coastguard Worker 76*387f9dfdSAndroid Build Coastguard WorkerUSAGE message: 77*387f9dfdSAndroid Build Coastguard Worker 78*387f9dfdSAndroid Build Coastguard Worker# ./stacksnoop -h 79*387f9dfdSAndroid Build Coastguard Workerusage: stacksnoop [-h] [-p PID] [-s] [-v] function 80*387f9dfdSAndroid Build Coastguard Worker 81*387f9dfdSAndroid Build Coastguard WorkerTrace and print kernel stack traces for a kernel function 82*387f9dfdSAndroid Build Coastguard Worker 83*387f9dfdSAndroid Build Coastguard Workerpositional arguments: 84*387f9dfdSAndroid Build Coastguard Worker function kernel function name 85*387f9dfdSAndroid Build Coastguard Worker 86*387f9dfdSAndroid Build Coastguard Workeroptional arguments: 87*387f9dfdSAndroid Build Coastguard Worker -h, --help show this help message and exit 88*387f9dfdSAndroid Build Coastguard Worker -p PID, --pid PID trace this PID only 89*387f9dfdSAndroid Build Coastguard Worker -s, --offset show address offsets 90*387f9dfdSAndroid Build Coastguard Worker -v, --verbose print more fields 91*387f9dfdSAndroid Build Coastguard Worker 92*387f9dfdSAndroid Build Coastguard Workerexamples: 93*387f9dfdSAndroid Build Coastguard Worker ./stacksnoop ext4_sync_fs # print kernel stack traces for ext4_sync_fs 94*387f9dfdSAndroid Build Coastguard Worker ./stacksnoop -s ext4_sync_fs # ... also show symbol offsets 95*387f9dfdSAndroid Build Coastguard Worker ./stacksnoop -v ext4_sync_fs # ... show extra columns 96*387f9dfdSAndroid Build Coastguard Worker ./stacksnoop -p 185 ext4_sync_fs # ... only when PID 185 is on-CPU 97