1*3e777be0SXin LiFrequently asked questions 2*3e777be0SXin Li========================== 3*3e777be0SXin Li 4*3e777be0SXin LiThese are issues that have been seen when using the Arm NN Android NNAPI driver. The guidance here could be of interest to Android system integrators including the driver in an Android platform. 5*3e777be0SXin Li 6*3e777be0SXin LiProblems seen when running VTS tests through vts-tradefed 7*3e777be0SXin Li--------------------------------------------------------- 8*3e777be0SXin Li 9*3e777be0SXin LiAn issue has been seen in some systems when running the NNAPI VTS tests using vts-tradefed, in a system configured to use the Arm NN GPU backend. 10*3e777be0SXin LiWhen the total time taken to run all tests in a suite exceeds the timeout defined for the plan, an error is supposed to be reported through vts-tradefed. However, 11*3e777be0SXin Lisome users have seen a situation where the device side of the tests gets killed, and the vts-tradefed console continues to run and reports that all further 12*3e777be0SXin Litests "Fail". In this case the "Failed" tests are actually not being executed. 13*3e777be0SXin Li 14*3e777be0SXin LiThis has been seen when running large test suites using the Arm NN GPU backend (Mali OpenCL) as the time needed to compile hundreds or thousands of OpenCL kernels 15*3e777be0SXin Liwas not taken into account when setting an appropriate test timeout for Android 8.1 and Android 9.0. This issue was fixed in the AOSP master branch in change 16*3e777be0SXin Lihttps://android.googlesource.com/platform/test/vts-testcase/hal/+/f74899c6c09b52703e6db0323dffb4ae52539db4 so should not be seen in Android 10 or later. 17*3e777be0SXin Li 18*3e777be0SXin LiAn acceptable workaround is to increase the timeout defined in AndroidTest.xml, in a similar way to https://android.googlesource.com/platform/test/vts-testcase/hal/+/f74899c6c09b52703e6db0323dffb4ae52539db4. 19*3e777be0SXin Li 20*3e777be0SXin LiInstance Normalization test failures 21*3e777be0SXin Li------------------------------------ 22*3e777be0SXin Li 23*3e777be0SXin LiThere is a known issue in the Android NNAPI implementation of Instance Normalization that has been verified as fixed from Android 10 r39 onwards. Using the Arm NN Android NNAPI driver with versions of the Android 10 VTS and CTS tests that do not have that fix will generate multiple Instance Normalization test failures. 24*3e777be0SXin Li 25*3e777be0SXin LiVTS and CTS test failures 26*3e777be0SXin Li------------------------- 27*3e777be0SXin Li 28*3e777be0SXin LiWith Android 10 R2 CTS some errors and crashes were discovered in the 19.08 and 19.11 releases of armnn, the android-nn-driver and ComputeLibrary. 19.08.01 and 19.11.01 releases of armnn, the android-nn-driver and ComputeLibrary were prepared that fix all these issues on CpuAcc and GpuAcc. If using 19.08 or 19.11 we recommend that you upgrade to the latest releases. 29*3e777be0SXin Li 30*3e777be0SXin LiThese fixes also required patches to be made to the Android Q test framework. You may encounter CTS and VTS test failures when attempting to build copies of the android-nn-driver against older versions of Android Q. 31*3e777be0SXin Li 32*3e777be0SXin LiThese test failures include: 33*3e777be0SXin Li 34*3e777be0SXin Li* ComputeMode/GeneratedTests.avg_pool_v1_2 Float16 tests 35*3e777be0SXin Li* ComputeMode/GeneratedTests.instance_normalization tests 36*3e777be0SXin Li* TestRandomGraph/SingleOperationTest.INSTANCE_NORMALIZATION_V1_2 tests 37*3e777be0SXin Li* TestRandomGraph/SingleOperationTest.PRELU_V1_2 tests 38*3e777be0SXin Li* Some TestRandomGraph/RandomGraphTest tests which include avg_pool or instance_normalization operators. 39*3e777be0SXin Li* Some TestRandomGraph/RandomGraphTest tests which use Float16 input. 40*3e777be0SXin Li 41*3e777be0SXin LiIn order to fix these failures you will have to update to a version of Android Q that includes the following patches: https://android-review.googlesource.com/q/project:platform%252Fframeworks%252Fml+branch:android10-tests-dev+status:merged 42*3e777be0SXin Li 43*3e777be0SXin LiThe Android 10 R3 CTS that can be downloaded from https://source.android.com/compatibility/cts/downloads contains all these patches. 44*3e777be0SXin Li 45*3e777be0SXin LiThere is a known issue that even with these patches CTS tests "TestRandomGraph/RandomGraphTest#LargeGraph_TENSOR_FLOAT16_Rank3/41" and "TestRandomGraph/RandomGraphTest#LargeGraph_TENSOR_FLOAT16_Rank2/20 " will still fail on CpuRef. These failures are caused by a LogSoftmax layer followed by a Floor layer which blows up the slight difference between fp16 to fp32. This issue only affects CpuRef with Android Q. These tests are not failing for Android R. 46