README.md
1# Profiler Overview
2
3This README describes the details of how the profiler is implemented.
4
5The profiler instruments PyTorch to collect information about the model's execution. Its main features are:
6* Instrumenting op calls on the CPU side
7* Interfacing with [Kineto](https://github.com/pytorch/kineto/) to collect information from the GPU (or other accelerators)
8* Collecting python stack traces
9* Exporting this information, e.g. in a chrome trace, or to be processed by downstream tools like [HTA](https://github.com/facebookresearch/HolisticTraceAnalysis)
10
11## Table of Contents
12
13- [Codebase Structure](#codebase-structure)
14- [`RecordFunction`](#recordfunction)
15- [Autograd Integration](#autograd-integration)
16- [Collection and Post-Processing](#collection-and-post-processing)
17- [Kineto Integration](#kineto-integration)
18- [Python Tracing](#python-tracing)
19
20## Codebase Structure ##
21
22TODO
23
24## `RecordFunction` ##
25
26[/aten/src/ATen/record_function.h](/aten/src/ATen/record_function.h)
27
28`RecordFunction` is used by the profiler to instrument CPU-side events.
29
30`RecordFunction` is a general method of instrumenting function calls in PyTorch. It can be used for other general applications, e.g. see [Features for Large-Scale Deployments](https://pytorch.org/docs/stable/notes/large_scale_deployments.html). In PyTorch, it is already included at some important locations; notably, in the [dispatcher](https://github.com/pytorch/pytorch/blob/247c603da9b780534e25fb1d90b6e5a528b625b1/aten/src/ATen/core/dispatch/Dispatcher.h#L650), surrounding every op.
31
32Users (or PyTorch itself) can register callbacks that will be executed whenever a `RecordFunction` guard is encountered. The profiler uses this mechanism to record the start and end times for each op call, as well as user-provided `RecordFunction` annotations. The `RecordFunction` machinery is designed to have relatively low overhead, especially when there are no callbacks registered. Nevertheless, there can still be some overhead.
33
34There is also a python binding for `RecordFunction` in python (`with torch.profiler.record_function`); this is often used by users to annotate events corresponding to module-level events.
35
36## Autograd Integration ##
37
38The autograd engine is responsible for automatically computing gradients.
39
40The profiler records two pieces of information from the autograd engine:
41* [Sequence number](/aten/src/ATen/SequenceNumber.h): this is a unique-per-thread index assigned to each op call(\*) in the forward pass. When a backward op is triggered, it is also assigned a sequence number matching the sequence number of the forward op that caused that backward op to be executed. Using this information, the profiler is able to match forward and backward ops; in chrome traces, this feature can be enabled with the "fwd_bwd" flow events
42* [Forward thread id](https://github.com/pytorch/pytorch/blob/2e3fce54506ba82eee2c890410bf7a1405a64ec6/aten/src/ATen/record_function.h#L357): Autograd can be used in multi-threaded environments. The forward thread ID indicates the ID of the thread on which the forward op was executed on. This information is needed because the sequence number, mentioned above, is only unique within a thread; the forward thread ID is used for differentiating different ops with the same sequence number.
43
44(\*) Note that only op invocations whose inputs require gradients are assigned a sequence number
45
46## Collection and Post-Processing ##
47
48TODO
49
50## Kineto Integration ##
51
52TODO
53
54## Python Tracing ##
55
56TODO
57