Lines Matching +full:fig +full:- +full:arch
14 "with open(\"benchmark-results.jsonl\") as f:\n",
25 "We have JSON Lines in [benchmark-results.jsonl](benchmark-results.jsonl) that look like this:"
42 " \"comment\": \"i9-9900k, 16GB DDR4-3000, Arch Linux\",\n",
44 " \"target_arch\": \"x86_64-pc-linux-gnu\"\n",
48 " \"cpu_model\": \"Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz\",\n",
52 " \"test-instr\": {\n",
85 " .dataframe tbody tr th:only-of-type {\n",
86 " vertical-align: middle;\n",
90 " vertical-align: top;\n",
94 " text-align: right;\n",
99 " <tr style=\"text-align: right;\">\n",
110 " <th>targets.test-instr.singlecore.execs_per_sec</th>\n",
112 " <th>targets.test-instr.singlecore.fuzzers_used</th>\n",
113 " <th>targets.test-instr-persist-shmem.singlecore.execs_per_sec</th>\n",
114 " <th>targets.test-instr-persist-shmem.singlecore.execs_total</th>\n",
115 " <th>targets.test-instr-persist-shmem.singlecore.fuzzers_used</th>\n",
116 " <th>targets.test-instr-persist-shmem.multicore.execs_per_sec</th>\n",
117 " <th>targets.test-instr-persist-shmem.multicore.execs_total</th>\n",
118 " <th>targets.test-instr-persist-shmem.multicore.fuzzers_used</th>\n",
119 " <th>targets.test-instr.multicore.execs_per_sec</th>\n",
120 " <th>targets.test-instr.multicore.execs_total</th>\n",
121 " <th>targets.test-instr.multicore.fuzzers_used</th>\n",
130 " <td>i9-9900k, 16GB DDR4-3000, Arch Linux</td>\n",
132 " <td>x86_64-pc-linux-gnu</td>\n",
134 " <td>Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz</td>\n",
154 " <td>i9-9900k, 16GB DDR4-3000, Arch Linux</td>\n",
156 " <td>x86_64-pc-linux-gnu</td>\n",
158 " <td>Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz</td>\n",
178 " <td>i9-9900k, 16GB DDR4-3000, Arch Linux</td>\n",
180 " <td>x86_64-pc-linux-gnu</td>\n",
182 " <td>Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz</td>\n",
202 " <td>i9-9900k, 16GB DDR4-3000, Arch Linux</td>\n",
204 " <td>x86_64-pc-linux-gnu</td>\n",
206 " <td>Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz</td>\n",
226 " <td>i9-9900k, 16GB DDR4-3000, Arch Linux</td>\n",
228 " <td>x86_64-pc-linux-gnu</td>\n",
230 " <td>Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz</td>\n",
259 "0 i9-9900k, 16GB DDR4-3000, Arch Linux clang version 16.0.6 \n",
260 "1 i9-9900k, 16GB DDR4-3000, Arch Linux clang version 16.0.6 \n",
261 "2 i9-9900k, 16GB DDR4-3000, Arch Linux clang version 16.0.6 \n",
262 "3 i9-9900k, 16GB DDR4-3000, Arch Linux clang version 16.0.6 \n",
263 "4 i9-9900k, 16GB DDR4-3000, Arch Linux clang version 16.0.6 \n",
266 "0 x86_64-pc-linux-gnu 4788.770 \n",
267 "1 x86_64-pc-linux-gnu 4989.281 \n",
268 "2 x86_64-pc-linux-gnu 4799.415 \n",
269 "3 x86_64-pc-linux-gnu 4703.293 \n",
270 "4 x86_64-pc-linux-gnu 4800.375 \n",
273 "0 Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz 16 \n",
274 "1 Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz 16 \n",
275 "2 Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz 16 \n",
276 "3 Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz 16 \n",
277 "4 Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz 16 \n",
279 " targets.test-instr.singlecore.execs_per_sec ... \\\n",
286 " targets.test-instr.singlecore.fuzzers_used \\\n",
293 " targets.test-instr-persist-shmem.singlecore.execs_per_sec \\\n",
300 " targets.test-instr-persist-shmem.singlecore.execs_total \\\n",
307 " targets.test-instr-persist-shmem.singlecore.fuzzers_used \\\n",
314 " targets.test-instr-persist-shmem.multicore.execs_per_sec \\\n",
321 " targets.test-instr-persist-shmem.multicore.execs_total \\\n",
328 " targets.test-instr-persist-shmem.multicore.fuzzers_used \\\n",
335 " targets.test-instr.multicore.execs_per_sec \\\n",
342 " targets.test-instr.multicore.execs_total \\\n",
349 " targets.test-instr.multicore.fuzzers_used \n",
376 …lecore and multicore modes, in each config setting -- where the x-axis is number of cores, and the…
387 "i7 = df.query(\"`config.comment` == 'i9-9900k, 16GB DDR4-3000, Arch Linux'\")\n",
400 …" max_fuzzers = int(query[[\"targets.test-instr-persist-shmem.multicore.fuzzers_used\", \"targe…
402 " for target in [\"test-instr-persist-shmem\", \"test-instr\"]:\n",
411 " if target == \"test-instr-persist-shmem\":\n",
416 " label += \"-multicore\"\n",
418 " label += \"-singlecore\"\n",
420 " label += \"+persist-conf\"\n",
422 " label += \"+system-conf\"\n",
424 " if label == \"shmem-multicore+persist-conf+system-conf\":\n",
426 " if label == \"shmem-multicore\":\n",
428 " if label == \"base-multicore+persist-conf+system-conf\":\n",
429 …onfig, \"afl_system_config\": afl_system_config, \"label\": \"Multicore: Non-persistent mode + ker…
430 " if label == \"shmem-singlecore+persist-conf+system-conf\":\n",
433 " if label == \"base-singlecore+persist-conf+system-conf\":\n",
435 …nfig, \"afl_system_config\": afl_system_config, \"label\": \"Singlecore: Non-persistent mode + ker…
449 …-svg\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"1…
463 "fig = pivotdf.plot(\n",
472 "# Compute tick values and their labels for the primary Y-axis\n",
475 "# Update the primary Y-axis with custom tick labels\n",
476 "fig.update_yaxes(tickvals=tickvals, ticktext=ticktext)\n",
477 "fig.update_xaxes(tickvals=list(range(1,36+1)))\n",
478 "fig.update_layout(width=1200, height=400)\n",
479 "fig.show(\"svg\")\n"
499 " .dataframe tbody tr th:only-of-type {\n",
500 " vertical-align: middle;\n",
504 " vertical-align: top;\n",
508 " text-align: right;\n",
513 " <tr style=\"text-align: right;\">\n",
515 " <th>Multicore: Non-persistent mode + kernel config</th>\n",
518 " <th>Singlecore: Non-persistent mode + kernel config</th>\n",
824 "label Multicore: Non-persistent mode + kernel config \\\n",
941 "label Singlecore: Non-persistent mode + kernel config \\\n",
1048 "#### test-instr vs. test-instr-persist-shmem\n",
1050 …"This graph is scaled so that the single-core, non-persistent-mode performance (9845 execs per sec…
1054 "#### Multicore test-instr\n",
1062 …Lplusplus/AFLplusplus/blob/stable/instrumentation/README.persistent_mode.md#4-persistent-mode),\n",
1063 …"we end up with **12.8x** base speed. So -- perhaps counter-intuively -- if you have a choice bet…
1064 …st on this machine) to use persistent mode on a single core, than to use non-persistent mode on al…
1073 …eferring to booting the Linux kernel with `mitigations=off`, which is a meta-parameter for disabli…
1074 ….2. Disabling these results in a `execs_per_sec` increase of 321386 execs -- the difference betwe…
1078 …"One way to think about this is that the mitigations turn this 16-thread CPU into a 7-thread CPU, …
1106 … "singlecore_base_execs = pivotdf.iloc[0][\"Singlecore: Non-persistent mode + kernel config\"]\n",
1109 … "multicore_base_max_execs = pivotdf[\"Multicore: Non-persistent mode + kernel config\"].max()\n",
1117 …"mitigations_off_increase = int(multicore_max_execs_mitigations_off - multicore_max_execs_mitigati…
1123 "#### test-instr vs. test-instr-persist-shmem\n",
1125 …"This graph is scaled so that the single-core, non-persistent-mode performance ({int(singlecore_ba…
1129 "#### Multicore test-instr\n",
1137 …Lplusplus/AFLplusplus/blob/stable/instrumentation/README.persistent_mode.md#4-persistent-mode),\n",
1138 …ctor_for_execs(singlecore_persist_execs)}x** base speed. So -- perhaps counter-intuively -- if yo…
1139 …st on this machine) to use persistent mode on a single core, than to use non-persistent mode on al…
1148 …eferring to booting the Linux kernel with `mitigations=off`, which is a meta-parameter for disabli…
1149 … results in a `execs_per_sec` increase of {mitigations_off_increase} execs -- the difference betwe…
1151 …"reduced the overall performance by {abs(round(((multicore_max_execs_mitigations_on - multicore_ma…
1153 …"One way to think about this is that the mitigations turn this 16-thread CPU into a 7-thread CPU, …
1166 …isingly only lose {abs(int(((multicore_max_execs_mitigations_off_only_cores - multicore_max_execs_…
1177 …"While there was some nuance here, the answer was pretty straightforward -- use the number of CPU …
1190 " .dataframe tbody tr th:only-of-type {\n",
1191 " vertical-align: middle;\n",
1195 " vertical-align: top;\n",
1199 " text-align: right;\n",
1204 " <tr style=\"text-align: right;\">\n",
1215 " <th>targets.test-instr-persist-shmem.multicore.execs_per_sec</th>\n",
1216 " <th>targets.test-instr-persist-shmem.multicore.execs_total</th>\n",
1217 " <th>targets.test-instr-persist-shmem.multicore.fuzzers_used</th>\n",
1227 " <td>clang version 15.0.7 (Amazon Linux 15.0.7-3.am...</td>\n",
1228 " <td>x86_64-amazon-linux-gnu</td>\n",
1242 " <td>clang version 15.0.7 (Amazon Linux 15.0.7-3.am...</td>\n",
1243 " <td>x86_64-amazon-linux-gnu</td>\n",
1265 "223 clang version 15.0.7 (Amazon Linux 15.0.7-3.am... \n",
1266 "224 clang version 15.0.7 (Amazon Linux 15.0.7-3.am... \n",
1269 "223 x86_64-amazon-linux-gnu 3514.326 \n",
1270 "224 x86_64-amazon-linux-gnu 3599.748 \n",
1276 " targets.test-instr-persist-shmem.multicore.execs_per_sec \\\n",
1280 " targets.test-instr-persist-shmem.multicore.execs_total \\\n",
1284 " targets.test-instr-persist-shmem.multicore.fuzzers_used \n",
1309 " .dataframe tbody tr th:only-of-type {\n",
1310 " vertical-align: middle;\n",
1314 " vertical-align: top;\n",
1318 " text-align: right;\n",
1323 " <tr style=\"text-align: right;\">\n",
1381 …-svg\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"1…
1399 "# Compute tick values and their labels for the primary Y-axis\n",
1402 "# Update the primary Y-axis with custom tick labels\n",
1415 …"This is a shocking result for a 192 vCPU machine -- our optimal number of parallel fuzzers was 16…
1417 …L++ is a bad fuzzer, or that AWS tricked us and gave us a 16-thread machine instead of a 192-threa…
1419 …-- the most likely cause here (other than a horrible bug) may be that we're already saturating the…
1435 "mimetype": "text/x-python",