1Minibench: ExecuTorch Android Benchmark App 2=== 3 4Minibench is a benchmarking app for testing the performance of the ExecuTorch runtime on Android devices. 5 6It supports both generic (vision, audio, etc) models and LLM. 7 8- For generic model, it reports metrics such as model load time, and average inference time. 9- For LLM, it reports metrics such as model load time, and tokens per second. 10- We are working on providing more metrics in the future. 11 12Minibench is usedful for giving reference performance data when developers integrate ExecuTorch with their own Android app. 13 14## Build 15You will need executorch AAR for Java and JNI dependencies. 16``` 17export ANDROID_NDK=<path_to_android_ndk> 18sh build/build_android_llm_demo.sh 19``` 20and copy the AAR to `app/libs`. 21``` 22mkdir -p app/libs 23cp $BUILD_AAR_DIR/executorch.aar app/libs 24``` 25 26You can also refer to [this script](https://github.com/pytorch/executorch/blob/62024d8/.github/workflows/android-perf.yml#L226-L235) to see how it is built. 27 28Then you can build and install the app on Android Studio, or simply run 29``` 30./gradlew installDebug 31``` 32 33## Usage 34This apk does not come with a launcher icon. Instead, trigger it from command line 35 36### Push model to a directory 37``` 38adb shell mkdir /data/local/tmp/minibench 39adb push my_model.pte /data/local/tmp/minibench 40# optionally, push tokenizer for LLM 41adb push tokenizer.bin /data/local/tmp/minibench 42``` 43 44### Generic model 45``` 46adb shell am start -W -S -n org.pytorch.minibench/org.pytorch.minibench.LlmBenchmarkActivity \ 47 --es model_dir /data/local/tmp/minibench 48``` 49 50### LLM 51``` 52adb shell am start -W -S -n org.pytorch.minibench/org.pytorch.minibench.LlmBenchmarkActivity \ 53 --es model_dir /data/local/tmp/minibench --es tokenizer_path /data/local/tmp/minibench/tokenizer.bin 54``` 55 56### Fetch results 57``` 58adb shell run-as org.pytorch.minibench cat files/benchmark_results.json 59``` 60If the ExecuTorch runner is initialized and loads your model, but there is a load error or run error, you will see error code from that JSON. 61