1# CLI Tool for Compile / Deploy Pre-Built QNN Artifacts 2 3An easy-to-use tool for generating / executing .pte program from pre-built model libraries / context binaries from Qualcomm AI Engine Direct. Tool is verified with [host environement](../../../../docs/source/build-run-qualcomm-ai-engine-direct-backend.md#host-os). 4 5## Description 6 7This tool aims for users who want to leverage ExecuTorch runtime framework with their existent artifacts generated by QNN. It's possible for them to produce .pte program in few steps.<br/> 8If users are interested in well-known applications, [Qualcomm AI HUB](https://aihub.qualcomm.com/) is a great approach which provides tons of optimized state-of-the-art models ready for deploying. All of them could be downloaded in model library or context binary format. 9 10* Model libraries(.so) came from `qnn-model-lib-generator` | AI HUB, or context binaries(.bin) came from `qnn-context-binary-generator` | AI HUB, could apply tool directly with: 11 - To produce .pte program: 12 ```bash 13 $ python export.py compile 14 ``` 15 - To perform inference with generated .pte program: 16 ```bash 17 $ python export.py execute 18 ``` 19 20### Dependencies 21 22* Register for Qualcomm AI HUB. 23* Download the corresponding QNN SDK via [link](https://www.qualcomm.com/developer/software/qualcomm-ai-engine-direct-sdk) which your favorite model is compiled with. Ths link will automatically download the latest version at this moment (users should be able to specify version soon, please refer to [this](../../../../docs/source/build-run-qualcomm-ai-engine-direct-backend.md#software) for earlier releases). 24 25### Target Model 26 27* Consider using [virtual environment](https://app.aihub.qualcomm.com/docs/hub/getting_started.html) for AI HUB scripts to prevent package conflict against ExecuTorch. Please finish the [installation section](https://app.aihub.qualcomm.com/docs/hub/getting_started.html#installation) before proceeding following steps. 28* Take [QuickSRNetLarge-Quantized](https://aihub.qualcomm.com/models/quicksrnetlarge_quantized?searchTerm=quantized) as an example, please [install](https://huggingface.co/qualcomm/QuickSRNetLarge-Quantized#installation) package as instructed. 29* Create workspace and export pre-built model library: 30 ```bash 31 mkdir $MY_WS && cd $MY_WS 32 # target chipset is `SM8650` 33 python -m qai_hub_models.models.quicksrnetlarge_quantized.export --target-runtime qnn --chipset qualcomm-snapdragon-8gen3 34 ``` 35* The compiled model library will be located under `$MY_WS/build/quicksrnetlarge_quantized/quicksrnetlarge_quantized.so`. This model library maps to the artifacts generated by SDK tools mentioned in `Integration workflow` section on [Qualcomm AI Engine Direct document](https://docs.qualcomm.com/bundle/publicresource/topics/80-63442-50/overview.html). 36 37### Compiling Program 38 39* Compile .pte program 40 ```bash 41 # `pip install pydot` if package is missing 42 # Note that device serial & hostname might not be required if given artifacts is in context binary format 43 PYTHONPATH=$EXECUTORCH_ROOT/.. python $EXECUTORCH_ROOT/examples/qualcomm/qaihub_scripts/utils/export.py compile -a $MY_WS/build/quicksrnetlarge_quantized/quicksrnetlarge_quantized.so -m SM8650 -s $DEVICE_SERIAL -b $EXECUTORCH_ROOT/build-android 44 ``` 45* Artifacts for checking IO information 46 - `output_pte/quicksrnetlarge_quantized/quicksrnetlarge_quantized.json` 47 - `output_pte/quicksrnetlarge_quantized/quicksrnetlarge_quantized.svg` 48 49### Executing Program 50 51* Prepare test image 52 ```bash 53 cd $MY_WS 54 wget https://user-images.githubusercontent.com/12981474/40157448-eff91f06-5953-11e8-9a37-f6b5693fa03f.png -O baboon.png 55 ``` 56 Execute following python script to generate input data: 57 ```python 58 import torch 59 import torchvision.transforms as transforms 60 from PIL import Image 61 img = Image.open('baboon.png').resize((128, 128)) 62 transform = transforms.Compose([transforms.PILToTensor()]) 63 # convert (C, H, W) to (N, H, W, C) 64 # IO tensor info. could be checked with quicksrnetlarge_quantized.json | .svg 65 img = transform(img).permute(1, 2, 0).unsqueeze(0) 66 torch.save(img, 'baboon.pt') 67 ``` 68* Execute .pte program 69 ```bash 70 PYTHONPATH=$EXECUTORCH_ROOT/.. python $EXECUTORCH_ROOT/examples/qualcomm/qaihub_scripts/utils/export.py execute -p output_pte/quicksrnetlarge_quantized -i baboon.pt -s $DEVICE_SERIAL -b $EXECUTORCH_ROOT/build-android 71 ``` 72* Post-process generated data 73 ```bash 74 cd output_data 75 ``` 76 Execute following python script to generate output image: 77 ```python 78 import io 79 import torch 80 import torchvision.transforms as transforms 81 # IO tensor info. could be checked with quicksrnetlarge_quantized.json | .svg 82 # generally we would have same layout for input / output tensors: e.g. either NHWC or NCHW 83 # this might not be true under different converter configurations 84 # learn more with converter tool from Qualcomm AI Engine Direct documentation 85 # https://docs.qualcomm.com/bundle/publicresource/topics/80-63442-50/tools.html#model-conversion 86 with open('output__142.pt', 'rb') as f: 87 buffer = io.BytesIO(f.read()) 88 img = torch.load(buffer, weights_only=False) 89 transform = transforms.Compose([transforms.ToPILImage()]) 90 img_pil = transform(img.squeeze(0)) 91 img_pil.save('baboon_upscaled.png') 92 ``` 93 You could check the upscaled result now! 94 95## Help 96 97Please check help messages for more information: 98```bash 99PYTHONPATH=$EXECUTORCH_ROOT/.. python $EXECUTORCH_ROOT/examples/qualcomm/aihub/utils/export.py -h 100PYTHONPATH=$EXECUTORCH_ROOT/.. python $EXECUTORCH_ROOT/examples/qualcomm/aihub/utils/python export.py compile -h 101PYTHONPATH=$EXECUTORCH_ROOT/.. python $EXECUTORCH_ROOT/examples/qualcomm/aihub/utils/python export.py execute -h 102``` 103
