## Inference benchmarks

This folder contains a work in progress simulation of a python inference server.

The v0 version of this has a backend worker that is a single process. It loads a
ResNet-18 checkpoint to 'cuda:0' and compiles the model. It accepts requests in
the form of (tensor, request_time) from a `multiprocessing.Queue`, runs
inference on the request and returns (output, request_time) in the a separate
response `multiprocessing.Queue`.

The frontend worker is a process with three threads
1. A thread that generates fake data of a given batch size in the form of CPU
   tensors and puts the data into the request queue
2. A thread that reads responses from the response queue and collects metrics on
   the latency of the first response, which corresponds to the cold start time,
   average, minimum and maximum response latency as well as throughput.
3. A thread that polls nvidia-smi for GPU utilization metrics.

For now we omit data preprocessing as well as result post-processing.

### Running a single benchmark

The togglable commmand line arguments to the script are as follows:
  - `num_iters` (default: 100): how many requests to send to the backend
    excluding the first warmup request
  - `batch_size` (default: 32): the batch size of the requests.
  - `model_dir` (default: '.'): the directory to load the checkpoint from
  - `compile` (default: compile): or `--no-compile` whether to `torch.compile()`
    the model
  - `output_file` (default: output.csv): The name of the csv file to write the outputs to in the `results/` directory.
  - `num_workers` (default: 2): The `max_threads` passed to the `ThreadPoolExecutor` in charge of model prediction

e.g. A sample command to run the benchmark

```
python -W ignore server.py --num_iters 1000 --batch_size 32
```

the results will be found in `results/output.csv`, which will be appended to if the file already exists.

Note that `m.compile()` time in the csv file is not the time for the model to be compiled,
which happens during the first iteration, but rather the time for PT2 components
to be lazily imported (e.g. triton).

### Running a sweep

The script `runner.sh` will run a sweep of the benchmark over different batch
sizes with compile on and off and collect the mean and standard deviation of warmup latency,
average latency, throughput and GPU utilization for each. The `results/` directory will contain the metrics
from running a sweep as we develop this benchmark where `results/output_{batch_size}_{compile}.md`
will contain the mean and standard deviation of results for a given batch size and compile setting.
If the file already exists, the metrics from the run will be appended as a new row in the markdown table.