1# Bionic Benchmarks 2 3[TOC] 4 5## libc benchmarks (bionic-benchmarks) 6 7`bionic-benchmarks` is a command line tool for measuring the runtimes of libc functions. It is built 8on top of [Google Benchmark](https://github.com/google/benchmark) with some additions to organize 9tests into suites. 10 11### Device benchmarks 12 13 $ mmma bionic/benchmarks 14 $ adb root 15 $ adb sync data 16 $ adb shell /data/benchmarktest/bionic-benchmarks/bionic-benchmarks 17 $ adb shell /data/benchmarktest64/bionic-benchmarks/bionic-benchmarks 18 19By default, `bionic-benchmarks` runs all of the benchmarks in alphabetical order. Pass 20`--benchmark_filter=getpid` to run just the benchmarks with "getpid" in their name. 21 22Note that we also build _static_ benchmark binaries. 23They're useful for testing on devices running different versions of Android, or running non-Android OSes. 24Those binaries are called `bionic-benchmarks-static` instead. 25Copy from `out/target/product/<device>/symbols/data/benchmarktest64/bionic-benchmarks-static` instead of 26`out/target/product/<device>/data/benchmarktest64/bionic-benchmarks-static` if you want symbols for perf(1). 27 28### Host benchmarks 29 30See the `benchmarks/run-on-host.sh` script. The host benchmarks can be run with 32-bit or 64-bit 31Bionic, or the host glibc. 32 33### XML suites 34 35Suites are stored in the `suites/` directory and can be chosen with the command line flag 36`--bionic_xml`. 37 38To choose a specific XML file, use the `--bionic_xml=FILE.XML` option. By default, this option 39searches for the XML file in the `suites/` directory. If it doesn't exist in that directory, then 40the file will be found as relative to the current directory. If the option specifies the full path 41to an XML file such as `/data/nativetest/suites/example.xml`, it will be used as-is. 42 43If no XML file is specified through the command-line option, the default is to use `suites/full.xml`. 44However, for the host bionic benchmarks (`bionic-benchmarks-glibc`), the default is to use 45`suites/host.xml`. 46 47### XML suite format 48 49The format for a benchmark is: 50 51``` 52<fn> 53 <name>BM_sample_benchmark</name> 54 <cpu><optional_cpu_to_lock></cpu> 55 <iterations><optional_iterations_to_run></iterations> 56 <args><space separated list of function args|shorthand></args> 57</fn> 58``` 59 60XML-specified values for iterations and cpu take precedence over those specified via command line 61(via `--bionic_iterations` and `--bionic_cpu`, respectively.) 62 63To make small changes in runs, you can also schedule benchmarks by passing in their name and a 64space-separated list of arguments via the `--bionic_extra` command line flag, e.g. 65`--bionic_extra="BM_string_memcpy AT_COMMON_SIZES"` or `--bionic_extra="BM_string_memcmp 32 8 8"` 66 67Note that benchmarks will run normally if extra arguments are passed in, and it will fail 68with a segfault if too few are passed in. 69 70### Shorthand 71 72For the sake of brevity, multiple runs can be scheduled in one XML element by putting one of the 73following in the args field: 74 75 NUM_PROPS 76 MATH_COMMON 77 AT_ALIGNED_<ONE|TWO>BUF 78 AT_<any power of two between 2 and 16384>_ALIGNED_<ONE|TWO>BUF 79 AT_COMMON_SIZES 80 81Definitions for these can be found in bionic_benchmarks.cpp, and example usages can be found in 82the suites directory. 83 84### Unit Tests 85 86`bionic-benchmarks` also has its own set of unit tests, which can be run from the binary in 87`/data/nativetest[64]/bionic-benchmarks-tests` 88 89## Process startup time (bionic-spawn-benchmarks) 90 91The `spawn/` subdirectory has a few benchmarks measuring the time used to start simple programs 92(e.g. Toybox's `true` and `sh -c true`). Run it on a device like so: 93 94 m bionic-spawn-benchmarks 95 adb root 96 adb sync data 97 adb shell /data/benchmarktest/bionic-spawn-benchmarks/bionic-spawn-benchmarks 98 adb shell /data/benchmarktest64/bionic-spawn-benchmarks/bionic-spawn-benchmarks 99 100Google Benchmark reports both a real-time figure ("Time") and a CPU usage figure. For these 101benchmarks, the CPU measurement only counts time spent in the thread calling `posix_spawn`, not that 102spent in the spawned process. The real-time is probably more useful, and it is the figure used to 103determine the iteration count. 104 105Locking the CPU frequency seems to improve the results of these benchmarks significantly, and it 106reduces variability. 107 108## Google Benchmark notes 109 110### Repetitions 111 112Google Benchmark uses two settings to control how many times to run each benchmark, "iterations" and 113"repetitions". By default, the repetition count is one. Google Benchmark runs the benchmark a few 114times to determine a sufficiently-large iteration count. 115 116Google Benchmark can optionally run a benchmark run repeatedly and report statistics (median, mean, 117standard deviation) for the runs. To do so, pass the `--benchmark_repetitions` option, e.g.: 118 119 # ./bionic-benchmarks --benchmark_filter=BM_stdlib_strtoll --benchmark_repetitions=4 120 ... 121 ------------------------------------------------------------------- 122 Benchmark Time CPU Iterations 123 ------------------------------------------------------------------- 124 BM_stdlib_strtoll 27.7 ns 27.7 ns 25290525 125 BM_stdlib_strtoll 27.7 ns 27.7 ns 25290525 126 BM_stdlib_strtoll 27.7 ns 27.7 ns 25290525 127 BM_stdlib_strtoll 27.8 ns 27.7 ns 25290525 128 BM_stdlib_strtoll_mean 27.7 ns 27.7 ns 4 129 BM_stdlib_strtoll_median 27.7 ns 27.7 ns 4 130 BM_stdlib_strtoll_stddev 0.023 ns 0.023 ns 4 131 132There are 4 runs, each with 25290525 iterations. Measurements for the individual runs can be 133suppressed if they aren't needed: 134 135 # ./bionic-benchmarks --benchmark_filter=BM_stdlib_strtoll --benchmark_repetitions=4 --benchmark_report_aggregates_only 136 ... 137 ------------------------------------------------------------------- 138 Benchmark Time CPU Iterations 139 ------------------------------------------------------------------- 140 BM_stdlib_strtoll_mean 27.8 ns 27.7 ns 4 141 BM_stdlib_strtoll_median 27.7 ns 27.7 ns 4 142 BM_stdlib_strtoll_stddev 0.043 ns 0.043 ns 4 143 144### CPU frequencies 145 146To get consistent results between runs, it can sometimes be helpful to restrict a benchmark to 147specific cores, or to lock cores at specific frequencies. Some phones have a big.LITTLE core setup, 148or at least allow some cores to run at higher frequencies than others. 149 150A core can be selected for `bionic-benchmarks` using the `--bionic_cpu` option or using the 151`taskset` utility. e.g. A Pixel 3 device has 4 Kryo 385 Silver cores followed by 4 Gold cores: 152 153 blueline:/ # /data/benchmarktest64/bionic-benchmarks/bionic-benchmarks --benchmark_filter=BM_stdlib_strtoll --bionic_cpu=0 154 ... 155 ------------------------------------------------------------ 156 Benchmark Time CPU Iterations 157 ------------------------------------------------------------ 158 BM_stdlib_strtoll 64.2 ns 63.6 ns 11017493 159 160 blueline:/ # /data/benchmarktest64/bionic-benchmarks/bionic-benchmarks --benchmark_filter=BM_stdlib_strtoll --bionic_cpu=4 161 ... 162 ------------------------------------------------------------ 163 Benchmark Time CPU Iterations 164 ------------------------------------------------------------ 165 BM_stdlib_strtoll 21.8 ns 21.7 ns 33167103 166 167A similar result can be achieved using `taskset`. The first parameter is a bitmask of core numbers 168to pass to `sched_setaffinity`: 169 170 blueline:/ # taskset f /data/benchmarktest64/bionic-benchmarks/bionic-benchmarks --benchmark_filter=BM_stdlib_strtoll 171 ... 172 ------------------------------------------------------------ 173 Benchmark Time CPU Iterations 174 ------------------------------------------------------------ 175 BM_stdlib_strtoll 64.3 ns 63.6 ns 10998697 176 177 blueline:/ # taskset f0 /data/benchmarktest64/bionic-benchmarks/bionic-benchmarks --benchmark_filter=BM_stdlib_strtoll 178 ... 179 ------------------------------------------------------------ 180 Benchmark Time CPU Iterations 181 ------------------------------------------------------------ 182 BM_stdlib_strtoll 21.3 ns 21.2 ns 33094801 183 184To lock the CPU frequency, use the sysfs interface at `/sys/devices/system/cpu/cpu*/cpufreq/`. 185Changing the scaling governor to `performance` suppresses the warning that Google Benchmark 186otherwise prints: 187 188 ***WARNING*** CPU scaling is enabled, the benchmark real time measurements may be noisy and will incur extra overhead. 189 190Some devices have a `perf-setup.sh` script that locks CPU and GPU frequencies. Some TradeFed 191benchmarks appear to be using the script. For more information: 192 * run `adb shell perf-setup.sh` to execute the script, it is already by default be installed on device for eng and userdebug build 193 * see: https://android.googlesource.com/platform/platform_testing/+/refs/heads/main/scripts/perf-setup/ 194