1## What is this? 2Python scripts in this folder are used to generate lite interpreter models for Android and iOS simulator tests. The goal of these tests is to detect changes that would break existing mobile models used in production (usually they are generated by earlier PyTorch versions). These scripts are based on PyTorch public API (https://pytorch.org/docs/stable/), and are grouped in a similar way: 3- math_ops (https://pytorch.org/docs/stable/torch.html#math-operations) 4 - pointwise_ops 5 - reduction_ops 6 - comparison_ops 7 - spectral_ops 8 - other_math_ops 9 - blas_lapack_ops 10- sampling_ops (https://pytorch.org/docs/stable/torch.html#random-sampling) 11- tensor ops (https://pytorch.org/docs/stable/torch.html#tensors) 12 - tensor_general_ops 13 - tensor_creation_ops 14 - tensor_indexing_ops 15 - tensor_typing_ops 16 - tensor_view_ops 17- nn ops (https://pytorch.org/docs/stable/nn.html) 18 - convolution_ops 19 - pooling_ops 20 - padding_ops 21 - activation_ops 22 - normalization_ops 23 - recurrent_ops 24 - transformer_ops 25 - linear_ops 26 - dropout_ops 27 - sparse_ops 28 - distance_function_ops 29 - loss_function_ops 30 - vision_function_ops 31 - shuffle_ops 32 - nn_utils_ops 33- quantization ops (https://pytorch.org/docs/stable/quantization.html) 34 - general_quant_ops 35 - dynamic_quant_ops 36 - static_quant_ops 37 - fused_quant_ops 38- TorchScript builtin ops (https://pytorch.org/docs/stable/jit_builtin_functions.html) 39 - torchscript_builtin_ops 40 - torchscript_collection_ops 41- torchvision_models (https://pytorch.org/vision/stable/models.html) 42 - mobilenet_v2 43 44The generated models are located at 45https://github.com/pytorch/pytorch/tree/master/android/pytorch_android/src/androidTest/assets (Android) 46https://github.com/pytorch/pytorch/tree/master/ios/TestApp/models/ (iOS) 47 48These test models will be executed in Android and iOS simulator tests. Note that we only check if there's error in model execution, but don't check the correctness of model output. 49 50## Checked-in models and on-the-fly models 51Each test model has a checked-in version and a on-the-fly version. The checked-in versions are stored in this repo (see above model paths) and will only be updated when necessary. The on-the-fly version will be generated during simulator test, with a "_temp" suffix, e.g., "reduction_ops_temp.ptl". Do not commit them. 52 53NOTE: currently Android simulator test does not generate on-the-fly models. Only iOS test does. 54 55## Diagnose failed test 56If the simulator test is falling, that means the current change will potentially break a production model. So be careful. The detailed error message can be found in test log. If the change has to be made, make sure it doesn't break existing production models, and update the failed test model as appropriate (see the next section). 57 58You can also run these tests locally, please see the instruction in android and ios folder. Remember to generate on-the-fly test models if you want to test it locally (but don't commit these models with _temp suffix). 59``` 60python test/mobile/model_test/gen_test_model.py ios-test 61``` 62 63## Update test model 64If for any reason a test model needs to be updated, run this script: 65``` 66python test/mobile/model_test/gen_test_model.py <model_name_without_suffix> 67``` 68For example, 69``` 70python test/mobile/model_test/gen_test_model.py reduction_ops 71python test/mobile/model_test/gen_test_model.py mobilenet_v2 72``` 73 74You can also update all test models for android and iOS: 75``` 76python test/mobile/model_test/gen_test_model.py android 77python test/mobile/model_test/gen_test_model.py ios 78``` 79 80## Test Coverage 81The test coverage is based on the number of root ops tested in these test models. The full list of generated ops can be found in: 82https://github.com/pytorch/pytorch/blob/master/test/mobile/model_test/coverage.yaml 83 84In additional, the simulator tests will also report the percentage of Meta's production ops that are covered. The list of production ops changes overtime, so a Meta employee needs to regularly udpate the list it using 85``` 86python test/mobile/model_test/update_production_ops.py ~/fbsource/xplat/pytorch_models/build/all_mobile_model_configs.yaml 87``` 88