README.md
1# `__torch_function__` micro-benchmarks
2
3This benchmark suite provides a systemic way to measure the performance of `__torch_function__` overhead.
4
5## Getting started
6### Initial Setup
7Install `py-spy` by doing:
8
9```bash
10pip install py-spy
11```
12
13Note that more extensive documentation on using `py-spy` is available in `CONTRIBUTING.md`.
14
15### Running the benchmark
16Run one of the following commands in the terminal, with the working directory being `${PYTORCH_CLONE_DIR}/benchmarks/overrides_benchmark`:
17
18```bash
19# Benchmark all the cases
20python bench.py
21
22# Flame graph pertaining to each case.
23py-spy record -o tensor.svg --native -- python pyspybench.py Tensor
24py-spy record -o subtensor.svg --native -- python pyspybench.py SubTensor
25py-spy record -o overridden.svg --native -- python pyspybench.py WithTorchFunction
26py-spy record -o suboverridden.svg --native -- python pyspybench.py SubWithTorchFunction
27```
28
29Here is a brief overview of what the results should look like, if run correctly:
30
31* Overhead for `torch` functions when run on `torch.Tensor` objects is on the order of 2 μs.
32* `__torch_function__` should add zero overhead for `torch.Tensor` inputs, a small overhead for subclasses of `torch.Tensor`, and a couple of microseconds for `Tensor`-likes with `__torch_function__`.
33* Changing the dispatching mechanism may result in changes that are on the order of 100 ns, which are hard to detect due to noise, but important.
34
35## Reporting benchmark results
36When modifying any of the machinery around `__torch_function__`, run the benchmark for both the feature branch and the point it diverges from `master`. For each of these:
37
38* Run `bench.py`, and include the output in your result.
39* For each case where `bench.py` shows a regression, run the commands described above, prefixing the output SVG filename (the input to the `-o` switch) with `base-` or `branch-` depending on the commit you are running the benchmark on.
40* For each SVG, open it in the browser, take a screenshot and include it in your result. Also include a ZIP file with all SVGs thus produced included.
41
