xref: /aosp_15_r20/external/google-fruit/extras/benchmark/README.md (revision a65addddcf69f38db5b288d787b6b7571a57bb8f)
1*a65addddSAndroid Build Coastguard Worker
2*a65addddSAndroid Build Coastguard Worker## Fruit benchmarks
3*a65addddSAndroid Build Coastguard Worker
4*a65addddSAndroid Build Coastguard WorkerFruit contains some code to benchmark various metrics (e.g. performance, compile time, executable size) in an automated
5*a65addddSAndroid Build Coastguard Workerway.
6*a65addddSAndroid Build Coastguard Worker
7*a65addddSAndroid Build Coastguard Worker### Benchmark suites
8*a65addddSAndroid Build Coastguard Worker
9*a65addddSAndroid Build Coastguard WorkerThe `suites` folder contains suites of Fruit (or Fruit-related) benchmarks that can be run using `run_benchmarks.py`.
10*a65addddSAndroid Build Coastguard WorkerFor example:
11*a65addddSAndroid Build Coastguard Worker
12*a65addddSAndroid Build Coastguard Worker```bash
13*a65addddSAndroid Build Coastguard Worker$ ~/projects/fruit/extras/benchmark/run_benchmarks.py \
14*a65addddSAndroid Build Coastguard Worker    --continue-benchmark=true \
15*a65addddSAndroid Build Coastguard Worker    --benchmark-definition ~/projects/fruit/extras/benchmark/suites/fruit_full.yml
16*a65addddSAndroid Build Coastguard Worker    --output-file ~/fruit_bench_results.txt \
17*a65addddSAndroid Build Coastguard Worker    --fruit-sources-dir ~/projects/fruit \
18*a65addddSAndroid Build Coastguard Worker    --fruit-benchmark-sources-dir ~/projects/fruit \
19*a65addddSAndroid Build Coastguard Worker    --boost-di-sources-dir ~/projects/boost-di
20*a65addddSAndroid Build Coastguard Worker```
21*a65addddSAndroid Build Coastguard Worker
22*a65addddSAndroid Build Coastguard WorkerOnce the benchmark run completes, you can format the results using some pre-defined tables, see the section below.
23*a65addddSAndroid Build Coastguard Worker
24*a65addddSAndroid Build Coastguard WorkerThe following benchmark suites are defined:
25*a65addddSAndroid Build Coastguard Worker
26*a65addddSAndroid Build Coastguard Worker* `fruit_full.yml`: full set of Fruit benchmarks (using the Fruit 3.x API).
27*a65addddSAndroid Build Coastguard Worker* `fruit_mostly_full.yml`: a subset of the tests in `fruit_full.yml`.
28*a65addddSAndroid Build Coastguard Worker* `fruit_quick.yml`: this is an even smaller subset, and the number of runs is capped at 10 so
29*a65addddSAndroid Build Coastguard Worker  the confidence intervals might be wider. It's useful as a quicker (around 10-15min) way to get a rough idea of the
30*a65addddSAndroid Build Coastguard Worker  performance (e.g. to evaluate the performance impact of a commit, during development).
31*a65addddSAndroid Build Coastguard Worker* `fruit_single.yml`: runs the Fruit runtime benchs under a single compiler and with just 1 combination of flags. This
32*a65addddSAndroid Build Coastguard Worker  also caps the number of runs at 8, so the resulting confidence intervals might be wider than they would be with
33*a65addddSAndroid Build Coastguard Worker  `fruit_full.yml`. This is a quick benchmark that can used during development of performance optimizations.
34*a65addddSAndroid Build Coastguard Worker* `fruit_debug.yml`: a suite used to debug Fruit's benchmarking code. This is very quick, but the actual results are
35*a65addddSAndroid Build Coastguard Worker  not meaningful. Run this after changing any benchmarking code, to check that it still works.
36*a65addddSAndroid Build Coastguard Worker* `boost_di`: unlike the others, this benchmark suite exercises the Boost.DI library (still in boost-experimental at the
37*a65addddSAndroid Build Coastguard Worker  time of writing) instead of Fruit.
38*a65addddSAndroid Build Coastguard Worker
39*a65addddSAndroid Build Coastguard Worker### Benchmark tables
40*a65addddSAndroid Build Coastguard Worker
41*a65addddSAndroid Build Coastguard WorkerThe `tables` folder contains some table definitions that can be used to get a human-readable representations of
42*a65addddSAndroid Build Coastguard Workerbenchmark results generated using `run_benchmarks.py`.
43*a65addddSAndroid Build Coastguard Worker
44*a65addddSAndroid Build Coastguard WorkerNote that there *isn't* a 1:1 mapping between benchmark suites and benchmark tables; the same table definition can be
45*a65addddSAndroid Build Coastguard Workerused with multiple benchmark suites (for example, a full suite and a quick variant that only has a subset of the
46*a65addddSAndroid Build Coastguard Workerdimensions) and multiple table definitions might be appropriate to display the results of a single suite (for example,
47*a65addddSAndroid Build Coastguard Workerthere could be a table definition that displays only metrics meaningful to Fruit users and one that also displays
48*a65addddSAndroid Build Coastguard Workermore fine-grained metrics that are only meaningful to Fruit developers).
49*a65addddSAndroid Build Coastguard Worker
50*a65addddSAndroid Build Coastguard WorkerExample usage of `format_bench_results.py` with one of these table definitions:
51*a65addddSAndroid Build Coastguard Worker
52*a65addddSAndroid Build Coastguard Worker```bash
53*a65addddSAndroid Build Coastguard Worker$ ~/projects/fruit/extras/benchmark/format_bench_results.py \
54*a65addddSAndroid Build Coastguard Worker    --benchmark-results ~/fruit_bench_results.txt \
55*a65addddSAndroid Build Coastguard Worker    --benchmark-tables-definition ~/projects/fruit/extras/benchmark/tables/fruit_wiki.yml
56*a65addddSAndroid Build Coastguard Worker```
57*a65addddSAndroid Build Coastguard Worker
58*a65addddSAndroid Build Coastguard WorkerIt's also possible to compare two benchmark results (for example, when running the same benchmarks before and after
59*a65addddSAndroid Build Coastguard Workera Fruit commit):
60*a65addddSAndroid Build Coastguard Worker
61*a65addddSAndroid Build Coastguard Worker```bash
62*a65addddSAndroid Build Coastguard Worker$ ~/projects/fruit/extras/benchmark/format_bench_results.py \
63*a65addddSAndroid Build Coastguard Worker    --benchmark-results ~/fruit_bench_results_after.txt \
64*a65addddSAndroid Build Coastguard Worker    --benchmark-tables-definition ~/projects/fruit/extras/benchmark/tables/fruit_wiki.yml \
65*a65addddSAndroid Build Coastguard Worker    --baseline-benchmark-results ~/fruit_bench_results_before.txt
66*a65addddSAndroid Build Coastguard Worker```
67*a65addddSAndroid Build Coastguard Worker
68*a65addddSAndroid Build Coastguard WorkerThe following tables are defined:
69*a65addddSAndroid Build Coastguard Worker
70*a65addddSAndroid Build Coastguard Worker* `fruit_wiki.yml`: the "main" table definition, with the tables that are in Fruit's wiki.
71*a65addddSAndroid Build Coastguard Worker* `fruit_internal.yml`: a more detailed version of `fruit_wiki.yml`, also displaying metrics that are only meaningful
72*a65addddSAndroid Build Coastguard Worker  to Fruit developers (e.g. splitting the setup time into component creation time and normalization time).
73*a65addddSAndroid Build Coastguard Worker
74*a65addddSAndroid Build Coastguard Worker### Manual benchmarks
75*a65addddSAndroid Build Coastguard Worker
76*a65addddSAndroid Build Coastguard WorkerIn some cases, you might want to run the benchmarks manually (e.g. if you want to use `callgrind` to profile the
77*a65addddSAndroid Build Coastguard Workerbenchmark run). This is how you can do that:
78*a65addddSAndroid Build Coastguard Worker
79*a65addddSAndroid Build Coastguard Worker```bash
80*a65addddSAndroid Build Coastguard Worker$ cd ~/projects/fruit
81*a65addddSAndroid Build Coastguard Worker$ mkdir build
82*a65addddSAndroid Build Coastguard Worker$ cd build
83*a65addddSAndroid Build Coastguard Worker$ CXX=g++-6 cmake .. -DCMAKE_BUILD_TYPE=RelWithDebInfo
84*a65addddSAndroid Build Coastguard Worker$ make -j 10
85*a65addddSAndroid Build Coastguard Worker$ cd ..
86*a65addddSAndroid Build Coastguard Worker$ mkdir generated-benchs
87*a65addddSAndroid Build Coastguard Worker$ extras/benchmark/generate_benchmark.py \
88*a65addddSAndroid Build Coastguard Worker    --compiler g++-6 \
89*a65addddSAndroid Build Coastguard Worker    --fruit-sources-dir ~/projects/fruit \
90*a65addddSAndroid Build Coastguard Worker    --fruit-build-dir ~/projects/fruit/build \
91*a65addddSAndroid Build Coastguard Worker    --num-components-with-no-deps 10 \
92*a65addddSAndroid Build Coastguard Worker    --num-components-with-deps 90 \
93*a65addddSAndroid Build Coastguard Worker    --num-deps 10 \
94*a65addddSAndroid Build Coastguard Worker    --output-dir generated-benchs \
95*a65addddSAndroid Build Coastguard Worker    --generate-debuginfo=true
96*a65addddSAndroid Build Coastguard Worker$ cd generated-benchs
97*a65addddSAndroid Build Coastguard Worker$ make -j 10
98*a65addddSAndroid Build Coastguard Worker$ valgrind \
99*a65addddSAndroid Build Coastguard Worker    --tool=callgrind \
100*a65addddSAndroid Build Coastguard Worker    --simulate-cache=yes \
101*a65addddSAndroid Build Coastguard Worker    --dump-instr=yes \
102*a65addddSAndroid Build Coastguard Worker    ./main 10000
103*a65addddSAndroid Build Coastguard Worker```
104