1## Batch Size 1 Compile false 2 3| Experiment | Warmup_latency (s) | Average_latency (s) | Throughput (samples/sec) | GPU Utilization (%) | 4| ---------- | ------------------ | ------------------- | ------------------------ | ------------------- | 5| original | 5.497 +/- 0.681 | 0.383 +/- 0.007 | 168.818 +/- 2.169 | 12.574 +/- 2.205 | 6| h2d_d2h_threads | 4.039 +/- 1.175 | 0.593 +/- 0.246 | 127.449 +/- 34.358 | 11.445 +/- 1.925 | 7| 2_predict_workers | 3.766 +/- 0.610 | 0.369 +/- 0.010 | 186.485 +/- 2.657 | 11.857 +/- 2.591 | 8| 3_predict_workers | 3.460 +/- 0.063 | 0.407 +/- 0.033 | 168.863 +/- 9.227 | 12.427 +/- 1.430 | 9| 4_predict_workers | 4.184 +/- 0.636 | 0.692 +/- 0.182 | 110.231 +/- 31.420 | 8.490 +/- 1.826 | 10