1### [#115286](https://github.com/pytorch/pytorch/pull/115286) 2* Prior to this PR, the backend worker was a process that read from the request queue, ran the model's forward and put the output in the response queue. In this PR, create a `ThreadPoolExecutor` with 1 worker and asynchronously run the model forward and response step in the executor so that it doesn't block polling the queue for more requests. 3 4##### Results 5* Warmup latency improved (likely due to the backend no longer being a new process) but all other metrics were worse. 6 7 8### [#116188](https://github.com/pytorch/pytorch/pull/116188) 9* Fixed two bugs in metrics calculation: 10 * Before this PR, each `request_time` was separated by the time for a `torch.randn(...)` to create the fake `data` tensor on CPU. This meant that the gap between requests incorrectly scaled with the batch size. Since the latency was calculated by `response_time - request_time`, the latencies were not comparable over different batch sizes. 11 * Corrected calculation of throughput: previously `(num_batches * batch_size) / sum(response_times)`, now `(num_batches * batch_size) / (last_response_time - first_request_time)` 12* Fixed bug where responses sent to frontend are on GPU. 13* Used a semaphore to ensure writing to `metrics_dict` in `metrics_thread` and `gpu_utilization_thread` in a thread-safe manner. 14 15##### Results 16* Baseline metrics were reset due to the bugs listed above. 17 18 19### [#116189](https://github.com/pytorch/pytorch/pull/116189) 20* Added two `ThreadPoolExecutor`s with 1 worker each for D2H and H2D copies. Each uses its own `cuda.Stream`. The purpose is to try to overlap D2H and H2D with compute and allow the worker handling prediction to launch compute kernels without being blocked by D2H/H2D. 21 * One thread pins memory of the CPU request and copies it into a CUDA tensor 22 * One thread moves the response to CPU and places it into the response queue 23Semaphores are used in conjunction with `cuda.Event`s to ensure proper synchronization among the threads. 24 25##### Results: 26* Warmup latency decreases as compared to the baseline for all batch sizes. 27* For batch sizes 1, 32, 64 we observed that metrics were worse 28 * Average latency increased 29 * Throughput decreased 30 * GPU utilization decreased 31* For batch sizes 128 and 256 we observed metrics improved 32 * Average latency decreased 33 * Throughput increased 34 * GPU utilization increased 35 36 37### [#116190](https://github.com/pytorch/pytorch/pull/116190) 38* Added a `--num_workers` option to `server.py` that allows more than 1 worker in the `ThreadPoolWorker` used for model predictions. Each worker uses its own `cuda.Stream()` that is created when the worker thread is initialized. 39 40##### Results: 41Benchmarks were only run for `compile=False` since `torch.compile()` is not thread-safe. Benchmarks were run with `num_workers={2, 3, 4}`. 42 43For the 2 worker case: 44* All metrics improved compared to the single worker case across all batch sizes. 45* For batch sizes 1, 32 and 64 we observed that the metrics were still slightly worse than the baseline. 46* For batch sizes 128 and 256 we observed that all metrics beat the baseline (e.g. ~300 samples/sec increase in throughput, ~5s decrease in average latency and ~2s decrease in warmup latency for bs=256) 47 48 49 50
