1## Batch Size 32 Compile false 2 3| Experiment | Warmup_latency (s) | Average_latency (s) | Throughput (samples/sec) | GPU Utilization (%) | 4| ---------- | ------------------ | ------------------- | ------------------------ | ------------------- | 5| original | 5.680 +/- 0.919 | 4.785 +/- 0.864 | 394.178 +/- 81.705 | 38.515 +/- 11.152 | 6| h2d_d2h_threads | 4.856 +/- 0.142 | 6.694 +/- 0.497 | 287.201 +/- 41.480 | 27.028 +/- 4.773 | 7| 2_predict_workers | 3.465 +/- 0.082 | 5.369 +/- 0.900 | 334.981 +/- 50.292 | 31.635 +/- 4.492 | 8| 3_predict_workers | 3.819 +/- 0.617 | 4.409 +/- 0.149 | 402.236 +/- 22.151 | 35.893 +/- 0.877 | 9| 4_predict_workers | 3.994 +/- 0.509 | 6.007 +/- 0.408 | 296.260 +/- 16.524 | 25.751 +/- 1.810 | 10