1*a65addddSAndroid Build Coastguard Worker 2*a65addddSAndroid Build Coastguard Worker## Fruit benchmarks 3*a65addddSAndroid Build Coastguard Worker 4*a65addddSAndroid Build Coastguard WorkerFruit contains some code to benchmark various metrics (e.g. performance, compile time, executable size) in an automated 5*a65addddSAndroid Build Coastguard Workerway. 6*a65addddSAndroid Build Coastguard Worker 7*a65addddSAndroid Build Coastguard Worker### Benchmark suites 8*a65addddSAndroid Build Coastguard Worker 9*a65addddSAndroid Build Coastguard WorkerThe `suites` folder contains suites of Fruit (or Fruit-related) benchmarks that can be run using `run_benchmarks.py`. 10*a65addddSAndroid Build Coastguard WorkerFor example: 11*a65addddSAndroid Build Coastguard Worker 12*a65addddSAndroid Build Coastguard Worker```bash 13*a65addddSAndroid Build Coastguard Worker$ ~/projects/fruit/extras/benchmark/run_benchmarks.py \ 14*a65addddSAndroid Build Coastguard Worker --continue-benchmark=true \ 15*a65addddSAndroid Build Coastguard Worker --benchmark-definition ~/projects/fruit/extras/benchmark/suites/fruit_full.yml 16*a65addddSAndroid Build Coastguard Worker --output-file ~/fruit_bench_results.txt \ 17*a65addddSAndroid Build Coastguard Worker --fruit-sources-dir ~/projects/fruit \ 18*a65addddSAndroid Build Coastguard Worker --fruit-benchmark-sources-dir ~/projects/fruit \ 19*a65addddSAndroid Build Coastguard Worker --boost-di-sources-dir ~/projects/boost-di 20*a65addddSAndroid Build Coastguard Worker``` 21*a65addddSAndroid Build Coastguard Worker 22*a65addddSAndroid Build Coastguard WorkerOnce the benchmark run completes, you can format the results using some pre-defined tables, see the section below. 23*a65addddSAndroid Build Coastguard Worker 24*a65addddSAndroid Build Coastguard WorkerThe following benchmark suites are defined: 25*a65addddSAndroid Build Coastguard Worker 26*a65addddSAndroid Build Coastguard Worker* `fruit_full.yml`: full set of Fruit benchmarks (using the Fruit 3.x API). 27*a65addddSAndroid Build Coastguard Worker* `fruit_mostly_full.yml`: a subset of the tests in `fruit_full.yml`. 28*a65addddSAndroid Build Coastguard Worker* `fruit_quick.yml`: this is an even smaller subset, and the number of runs is capped at 10 so 29*a65addddSAndroid Build Coastguard Worker the confidence intervals might be wider. It's useful as a quicker (around 10-15min) way to get a rough idea of the 30*a65addddSAndroid Build Coastguard Worker performance (e.g. to evaluate the performance impact of a commit, during development). 31*a65addddSAndroid Build Coastguard Worker* `fruit_single.yml`: runs the Fruit runtime benchs under a single compiler and with just 1 combination of flags. This 32*a65addddSAndroid Build Coastguard Worker also caps the number of runs at 8, so the resulting confidence intervals might be wider than they would be with 33*a65addddSAndroid Build Coastguard Worker `fruit_full.yml`. This is a quick benchmark that can used during development of performance optimizations. 34*a65addddSAndroid Build Coastguard Worker* `fruit_debug.yml`: a suite used to debug Fruit's benchmarking code. This is very quick, but the actual results are 35*a65addddSAndroid Build Coastguard Worker not meaningful. Run this after changing any benchmarking code, to check that it still works. 36*a65addddSAndroid Build Coastguard Worker* `boost_di`: unlike the others, this benchmark suite exercises the Boost.DI library (still in boost-experimental at the 37*a65addddSAndroid Build Coastguard Worker time of writing) instead of Fruit. 38*a65addddSAndroid Build Coastguard Worker 39*a65addddSAndroid Build Coastguard Worker### Benchmark tables 40*a65addddSAndroid Build Coastguard Worker 41*a65addddSAndroid Build Coastguard WorkerThe `tables` folder contains some table definitions that can be used to get a human-readable representations of 42*a65addddSAndroid Build Coastguard Workerbenchmark results generated using `run_benchmarks.py`. 43*a65addddSAndroid Build Coastguard Worker 44*a65addddSAndroid Build Coastguard WorkerNote that there *isn't* a 1:1 mapping between benchmark suites and benchmark tables; the same table definition can be 45*a65addddSAndroid Build Coastguard Workerused with multiple benchmark suites (for example, a full suite and a quick variant that only has a subset of the 46*a65addddSAndroid Build Coastguard Workerdimensions) and multiple table definitions might be appropriate to display the results of a single suite (for example, 47*a65addddSAndroid Build Coastguard Workerthere could be a table definition that displays only metrics meaningful to Fruit users and one that also displays 48*a65addddSAndroid Build Coastguard Workermore fine-grained metrics that are only meaningful to Fruit developers). 49*a65addddSAndroid Build Coastguard Worker 50*a65addddSAndroid Build Coastguard WorkerExample usage of `format_bench_results.py` with one of these table definitions: 51*a65addddSAndroid Build Coastguard Worker 52*a65addddSAndroid Build Coastguard Worker```bash 53*a65addddSAndroid Build Coastguard Worker$ ~/projects/fruit/extras/benchmark/format_bench_results.py \ 54*a65addddSAndroid Build Coastguard Worker --benchmark-results ~/fruit_bench_results.txt \ 55*a65addddSAndroid Build Coastguard Worker --benchmark-tables-definition ~/projects/fruit/extras/benchmark/tables/fruit_wiki.yml 56*a65addddSAndroid Build Coastguard Worker``` 57*a65addddSAndroid Build Coastguard Worker 58*a65addddSAndroid Build Coastguard WorkerIt's also possible to compare two benchmark results (for example, when running the same benchmarks before and after 59*a65addddSAndroid Build Coastguard Workera Fruit commit): 60*a65addddSAndroid Build Coastguard Worker 61*a65addddSAndroid Build Coastguard Worker```bash 62*a65addddSAndroid Build Coastguard Worker$ ~/projects/fruit/extras/benchmark/format_bench_results.py \ 63*a65addddSAndroid Build Coastguard Worker --benchmark-results ~/fruit_bench_results_after.txt \ 64*a65addddSAndroid Build Coastguard Worker --benchmark-tables-definition ~/projects/fruit/extras/benchmark/tables/fruit_wiki.yml \ 65*a65addddSAndroid Build Coastguard Worker --baseline-benchmark-results ~/fruit_bench_results_before.txt 66*a65addddSAndroid Build Coastguard Worker``` 67*a65addddSAndroid Build Coastguard Worker 68*a65addddSAndroid Build Coastguard WorkerThe following tables are defined: 69*a65addddSAndroid Build Coastguard Worker 70*a65addddSAndroid Build Coastguard Worker* `fruit_wiki.yml`: the "main" table definition, with the tables that are in Fruit's wiki. 71*a65addddSAndroid Build Coastguard Worker* `fruit_internal.yml`: a more detailed version of `fruit_wiki.yml`, also displaying metrics that are only meaningful 72*a65addddSAndroid Build Coastguard Worker to Fruit developers (e.g. splitting the setup time into component creation time and normalization time). 73*a65addddSAndroid Build Coastguard Worker 74*a65addddSAndroid Build Coastguard Worker### Manual benchmarks 75*a65addddSAndroid Build Coastguard Worker 76*a65addddSAndroid Build Coastguard WorkerIn some cases, you might want to run the benchmarks manually (e.g. if you want to use `callgrind` to profile the 77*a65addddSAndroid Build Coastguard Workerbenchmark run). This is how you can do that: 78*a65addddSAndroid Build Coastguard Worker 79*a65addddSAndroid Build Coastguard Worker```bash 80*a65addddSAndroid Build Coastguard Worker$ cd ~/projects/fruit 81*a65addddSAndroid Build Coastguard Worker$ mkdir build 82*a65addddSAndroid Build Coastguard Worker$ cd build 83*a65addddSAndroid Build Coastguard Worker$ CXX=g++-6 cmake .. -DCMAKE_BUILD_TYPE=RelWithDebInfo 84*a65addddSAndroid Build Coastguard Worker$ make -j 10 85*a65addddSAndroid Build Coastguard Worker$ cd .. 86*a65addddSAndroid Build Coastguard Worker$ mkdir generated-benchs 87*a65addddSAndroid Build Coastguard Worker$ extras/benchmark/generate_benchmark.py \ 88*a65addddSAndroid Build Coastguard Worker --compiler g++-6 \ 89*a65addddSAndroid Build Coastguard Worker --fruit-sources-dir ~/projects/fruit \ 90*a65addddSAndroid Build Coastguard Worker --fruit-build-dir ~/projects/fruit/build \ 91*a65addddSAndroid Build Coastguard Worker --num-components-with-no-deps 10 \ 92*a65addddSAndroid Build Coastguard Worker --num-components-with-deps 90 \ 93*a65addddSAndroid Build Coastguard Worker --num-deps 10 \ 94*a65addddSAndroid Build Coastguard Worker --output-dir generated-benchs \ 95*a65addddSAndroid Build Coastguard Worker --generate-debuginfo=true 96*a65addddSAndroid Build Coastguard Worker$ cd generated-benchs 97*a65addddSAndroid Build Coastguard Worker$ make -j 10 98*a65addddSAndroid Build Coastguard Worker$ valgrind \ 99*a65addddSAndroid Build Coastguard Worker --tool=callgrind \ 100*a65addddSAndroid Build Coastguard Worker --simulate-cache=yes \ 101*a65addddSAndroid Build Coastguard Worker --dump-instr=yes \ 102*a65addddSAndroid Build Coastguard Worker ./main 10000 103*a65addddSAndroid Build Coastguard Worker``` 104