1## TestApp 2 3The TestApp is currently being used as a dummy app by Circle CI for nightly jobs. The challenge comes when testing the arm64 build as we don't have a way to code-sign our TestApp. This is where Fastlane came to rescue. [Fastlane](https://fastlane.tools/) is a trendy automation tool for building and managing iOS applications. It also works seamlessly with Circle CI. We are going to leverage the `import_certificate` action, which can install developer certificates on CI machines. See `Fastfile` for more details. 4 5For simulator build, we run unit tests as the last step of our CI workflow. Those unit tests can also be run manually via the `fastlane scan` command. 6 7## Run Simulator Test Locally 8Follow these steps if you want to run the test locally. 9 101. Checkout PyTorch repo including all submodules 11 122. Build PyTorch for ios 13``` 14USE_COREML_DELEGATE=1 IOS_PLATFORM=SIMULATOR ./scripts/build_ios.sh 15``` 16 173. Generate on-the-fly test models 18``` 19python test/mobile/model_test/gen_test_model.py ios-test 20``` 21You need to install regular PyTorch on your local machine to run this script. 22Check https://github.com/pytorch/pytorch/tree/master/test/mobile/model_test#diagnose-failed-test to learn more. 23 244. Create XCode project (for lite interpreter) 25``` 26cd ios/TestApp/benchmark 27ruby setup.rb --lite 1 28``` 29 305. Open the generated TestApp/TestApp.xcodeproj in XCode and run simulator test. 31 32## Re-generate All Test Models 331. Make sure PyTorch (not PyTorch for iOS) is installed 34See https://pytorch.org/get-started/locally/ 35 362. Re-generate models for operator test 37``` 38python test/mobile/model_test/gen_test_model.py ios 39python test/mobile/model_test/gen_test_model.py ios-test 40``` 41 423. Re-generate Core ML model 43``` 44cd ios/TestApp/benchmark; python coreml_backend.py 45``` 46 47## Run test on AWS Device Farm 48The test app and its test suite could also be run on actual devices via 49AWS Device Farm. 50 511. The following steps could only be done on MacOS with Xcode installed. 52 I'm using Xcode 15.0 on MacOS M1 arm64 53 542. Checkout PyTorch repo including all submodules 55 563. Build PyTorch for iOS devices, not for simulator 57``` 58export BUILD_LITE_INTERPRETER=1 59export USE_PYTORCH_METAL=1 60export USE_COREML_DELEGATE=1 61export IOS_PLATFORM=OS 62export IOS_ARCH=arm64 63 64./scripts/build_ios.sh 65``` 66 674. Build the test app locally 68``` 69# Use the pytorch nightly build to generate models 70pip install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cpu 71 72# Generate models for differnet backends 73pushd ios/TestApp/benchmark 74mkdir -p ../models 75 76# This requires numpy==1.23.1 77python coreml_backend.py 78 79# NB: Also need to set the team ID with -t if you are running this locally. This 80# command setups an app that could be used to launch TestAppTests on device. On 81# the other hand, adding the --benchmark flag to build the one that runs benchmark 82# instead. 83ruby setup.rb --lite 1 84popd 85 86# Build the TestApp and its TestAppTests 87ruby scripts/xcode_build.rb -i build_ios/install -x ios/TestApp/TestApp.xcodeproj -p "OS" 88``` 89 905. Prepare the artifacts 91https://docs.aws.amazon.com/devicefarm/latest/developerguide/test-types-ios-xctest.html 92 93``` 94export DEST_DIR="Payload" 95 96pushd ios/TestApp/build/Release-iphoneos 97mkdir "${DEST_DIR}" 98 99cp -r TestApp.app "${DEST_DIR}" 100# TestApp.ipa is just a zip file with a payload subdirectory 101zip -vr TestApp.ipa "${DEST_DIR}" 102 103pushd TestApp.app/PlugIns 104# Also zip the TestAppTests.xctest test suite 105zip -vr TestAppTests.xctest.zip TestAppTests.xctest 106popd 107 108cp TestApp.app/PlugIns/TestAppTests.xctest.zip . 109popd 110``` 111 1126. Upload the artifacts to AWS Device Farm and run the tests 113``` 114export PYTORCH_ARN="arn:aws:devicefarm:us-west-2:308535385114:project:b531574a-fb82-40ae-b687-8f0b81341ae0" 115 116pushd ios/TestApp 117# AWS Device Farm is only available on us-west-2 118AWS_DEFAULT_REGION=us-west-2 python run_on_aws_devicefarm.py \ 119 --project-arn "${PYTORCH_ARN}" \ 120 --app-file build/Release-iphoneos/TestApp.ipa \ 121 --xctest-file build/Release-iphoneos/TestAppTests.xctest.zip \ 122 --name-prefix PyTorch 123popd 124``` 125 1267. The script will continue polling for the outcome. A visual output of 127 the test results could be view on AWS Device Farm console for [PyTorch project](https://us-west-2.console.aws.amazon.com/devicefarm/home#/mobile/projects/b531574a-fb82-40ae-b687-8f0b81341ae0/runs) 128 129## Debug Test Failures 130Make sure all models are generated. See https://github.com/pytorch/pytorch/tree/master/test/mobile/model_test to learn more. 131 132There's no debug information in simulator test (project TestAppTests). You can copy the failed test code to 133TestApp/TestApp/ViewController.mm and debug in the main TestApp. 134 135### Benchmark 136 137The benchmark folder contains two scripts that help you setup the benchmark project. The `setup.rb` does the heavy-lifting jobs of setting up the XCode project, whereas the `trace_model.py` is a Python script that you can tweak to generate your model for benchmarking. Simply follow the steps below to setup the project 138 1391. In the PyTorch root directory, run `IOS_ARCH=arm64 ./scripts/build_ios.sh` to generate the custom build from **Master** branch 1402. Navigate to the `benchmark` folder, run `python trace_model.py` to generate your model. 1413. In the same directory, open `config.json`. Those are the input parameters you can tweak. 1424. Again, in the same directory, run `ruby setup.rb` to setup the XCode project. 1435. Open the `TestApp.xcodeproj`, you're ready to go. 144 145The benchmark code is written in C++, you can use `UI_LOG` to visualize the log. See `benchmark.mm` for more details. 146