1VTS LTP Workflow 2================ 3 4A Guide to Developing LTP for VTS/Android 5 6What is LTP? 7------------ 8 9[Github](https://github.com/linux-test-project/ltp) (upstream) 10 11LTP (Linux Test Project) is a suite of tests that covers both kernel interfaces 12and userspace functionality (glibc, commonly used binaries, etc). For the 13purposes of Android the userspace functionality testing is of less importance 14and in fact much of it must be disabled, given the functionality is not 15available in Android. 16 17As of Jan 2023 there are on the order of 1300 tests executed in VTS on arm64. 18Most tests are run in both 32-bit and 64-bit mode. Many more are available but 19currently disabled due to either being broken or not applicable on Android. 20 21How is LTP Run in VTS? 22---------------------- 23 24The LTP source is located at external/ltp in the Android tree. This is not an 25exact mirror of upstream, there are outstanding changes to LTP for it to work 26on Android which have not yet been pushed upstream. In addition to the LTP 27source there is also the VTS wrapper around it. This is located at 28test/vts-testcase/kernel/ltp. Some noteworthy directories/files: 29 30* `external/ltp/android/`: Contains Android-specific files, aside from Android.[bp, mk] at top level. 31* `external/ltp/android/Android.ltp.mk`: Lists build rules for the LTP modules built under make. This file gets auto-generated by android/tools/gen_android_build.sh. 32* `external/ltp/gen.bp`: Lists build rules for the LTP modules built under Soong. This file gets auto-generated by android/tools/gen_android_build.sh. 33* `external/ltp/android/ltp_package_list.mk`: Lists all tests that will get pulled into VTS - VTS depends on this list. This file gets auto-generated by android/tools/gen_android_build.sh. 34* `external/ltp/android/tools/disabled_tests.txt`: Lists tests which cannot or should not be compiled for Android. This file is read by gen_android_build.sh during LTP upgrades to produce *.mk files required to build LTP for Android. 35* `external/ltp/testcases`: Source for LTP tests. Among the most important for the purposes of Treble are those in external/ltp/testcases/kernel/syscalls. 36* `test/vts-testcase/kernel/ltp/testcase/tools/configs/disabled_tests.py`: Any test listed here will not be run in VTS, despite being compiled. 37* `test/vts-testcase/kernel/ltp/testcase/tools/configs/stable_tests.py`: Any test listed here will run as part of the vts_ltp_test_arm/vts_ltp_test_arm_64 modules. 38 39Running LTP through atest 40------------------------- 41You can run LTP tests with atest, which handles all the setup and build steps. 42 43To run all 32 bit LTP tests: 44* `atest vts_ltp_test_arm` 45 46To run all 64 bit LTP tests: 47* `atest vts_ltp_test_arm_64` 48 49To run a single test: 50* `atest vts_ltp_test_arm:dio.dio13_32bit` 51 52Running LTP through VTS 53----------------------- 54To run VTS LTP it must first be built. VTS is not device specific, you need not 55compile it specifically for the device you wish to run it on, assuming it is 56the same architecture. 57 58* `. build/envsetup.sh` 59* `lunch` 60* `make -j vts` 61 62Then open vts-tradefed and run the VTS stable set: 63* `vts-tradefed` 64* `vts-tf > run vts-kernel -m vts_ltp_test_arm` 65 66If you regularly work with multiple devices it may be useful to specify the 67specific device you wish to run VTS on via the serial number: 68* `vts-tf > run vts-kernel -m vts_ltp_test_arm -s 000123456789` 69 70Or a specific test within the stable set: 71* `vts-tf > run vts-kernel -m vts_ltp_test_arm -t dio.dio13_32bit` 72 73Running LTP Directly 74-------------------- 75 76Running LTP tests within VTS can be quite cumbersome, especially if you are 77iterating a lot trying to debug something. Build and run LTP tests faster by 78doing 79 80* `external/ltp$ mma` 81* `external/ltp$ adb sync data` 82 83The test cases will be located at /data/nativetest{64,}/ltp/testcases/bin. 84 85Sometimes you need to perform this step after syncing: 86* `external/ltp$ git clean -x -f -d` 87Otherwise, build will fail. 88 89In order to simulate the exact environment that VTS will be creating for each 90of these tests, you can set the following environment variables before you run 91the test. This is very useful if the test itself depends on some of these 92variables to be set appropriately. 93 94* `adb root && adb shell` 95 96In the root shell on device: 97* `mkdir -p /data/local/tmp/ltp/tmp/ltptemp` 98* `mkdir -p /data/local/tmp/ltp/tmp/tmpbase` 99* `mkdir -p /data/local/tmp/ltp/tmp/tmpdir` 100* `restorecon -F -R /data/local/tmp/ltp` 101* `export TMP=/data/local/tmp/ltp/tmp` 102* `export LTPTMP=/data/local/tmp/ltp/tmp/ltptemp` 103* `export LTP_DEV_FS_TYPE=ext4` 104* `export TMPBASE=/data/local/tmp/ltp/tmp/tmpbase` 105* `export TMPDIR=/data/local/tmp/ltp/tmp/tmpdir` 106* `export LTPROOT=/data/local/tmp/ltp` 107 108For running 64-bit tests: 109* `export PATH=/data/nativetest64/ltp/testcases/bin:$PATH` 110 111Or For running 32-bit tests: 112* `export PATH=/data/nativetest/ltp/testcases/bin:$PATH` 113 114How do I enable or disable tests from the LTP build? 115---------------------------------------------------- 116 117Tests are disabled from the LTP build by adding them to 118external/ltp/android/tools/disabled_tests.txt. Many tests have been added to 119this file over time. Some of them are not applicable to Android and therefore 120should not be built. Others were disabled here because they were failing at one 121point in time for reasons unknown. 122 123To make a change to what is built in LTP, add or remove an entry in this file, 124and then update the Android-specific build files for LTP, mentioned above: 125 126* `external/ltp/android/Android.ltp.mk`, for LTP modules built in make 127* `external/ltp/gen.bp`, for LTP modules built in soong 128* `external/ltp/android/ltp_package_list.mk`, which lists all LTP modules that get pulled into VTS 129 130Update these files by running the script located at 131external/ltp/android/tools/gen_android_build.sh. Instructions for the script 132are in external/ltp/android/how-to-update.txt. 133 134 135How do I enable or disable tests from VTS LTP? 136---------------------------------------------- 137 138In addition to whether modules are built in external/ltp, it is also necessary 139to configure the VTS harness for LTP to determine which tests are in the stable 140set, the staging set, or disabled. Note that being disabled in VTS does not 141affect whether the test is built, but rather determines whether it is run at 142all as part of the stable or staging sets. 143 144The file test/vts-testcase/kernel/ltp/testcase/tools/configs/stable_tests.py 145lists tests that will run as part of VTS (vts_ltp_test_arm/vts_ltp_test_arm_64). 146 147When a test is enabled for the first time in VTS it should be in the staging 148set. The behavior of the test will be observed over a period of time and ensure 149the test is reliable. After a period of time (a week or two) it will be moved 150to the stable set. 151 152Tests that will never be relevant to Android should be disabled from the build 153in external/ltp. Tests that are (hopefully) temporarily broken should be 154runtime disabled in VTS. The staging and stable sets should normally all be 155passing. If something is failing there it should either be fixed with priority 156or disabled until it can be fixed. 157 158If the runtime of LTP changes significantly be sure to update the runtime-hint 159and test-timeout parameters to VTS in 160`test/vts-testcase/kernel/ltp/stable/AndroidTest.xml`. 161 162 163How do I see recent VTS LTP results? 164---------------------------------------------------------- 165 166The internal portal at go/vts11-dashboard shows results for the continuous VTS testing 167done on internal devices. 168 169Test results are also gathered by Linaro and may be seen 170[here](https://qa-reports.linaro.org/android-lkft/). 171 172 173Help! The external/ltp build is failing! 174---------------------------------------- 175 176Try doing a make distclean inside of external/ltp. If an upgrade to LTP has 177recently merged or the build files were recently updated, stale files in 178external/ltp can cause build failures. 179 180 181What outstanding issues exist? 182------------------------------ 183 184The hotlist for LTP bugs is [ltp-todo](https://buganizer.corp.google.com/hotlists/733268). 185 186When you begin working on an LTP bug please assign the bug to yourself so that 187others know it is being worked on. 188 189 190Testing x86_64 191-------------- 192 193It is not advisable to run LTP tests directly on your host unless you are fully 194aware of what the tests will do and are okay with it. These tests may 195destabilize your box or cause data loss. If you need to run tests on an x86 196platform and are unsure if they are safe you should run them in emulation, in a 197virtualized environment, or on a dedicated development x86 platform. 198 199To run LTP tests for x86 platform, you can do: 200* `atest vts_ltp_test_x86` 201* `atest vts_ltp_test_x86_64` 202 203 204Sending Fixes Upstream 205---------------------- 206 207The mailing list for LTP is located 208[here](https://lists.linux.it/listinfo/ltp). Some standard kernel guidelines 209apply to sending patches; they should be checkpatch (scripts/checkpatch.pl in 210the kernel repository) clean and sent in plain text in canonical patch format. 211One easy way to do this is by using git format-patch and git send-email. 212 213There is an #LTP channel on freenode. The maintainer Cyril Hrubis is there (his 214nick is metan). 215 216 217Merging Fixes 218------------------------ 219 220When possible please merge fixes upstream first. Then cherrypick the change 221onto aosp/master in external/ltp. 222 223 224Upgrade LTP to the latest upstream release 225------------------------------------------- 226 227LTP has three releases per year. Keeping the current project aligned with the 228upstream development is important to get additional tests and bug-fixes. 229 230### Merge the changes 231 232AOSP external projects have a branch that track the changes to the upstream 233repository, called `aosp/upstream-master`. 234That branch is automatically updated with: 235 236`repo sync .` 237 238Create a new branch to work on the merge, that will contain the merge commit 239itself and conflicts resolutions: 240 241`repo start mymerge .` 242 243Find the commit for the latest LTP release, for example 244 245``` 246$ git log --oneline aosp/upstream-master 247c00f96994 (aosp/upstream-master) openposix/Makefile: Use tabs instead of spaces 248a90664f8d Makefile: Use SPDX in Makefile 2490fb171f2b LTP 20210524 250``` 251 252Force the creation of a merge commit (no fast-forward). 253 254`git merge <release commit> --no-ff` 255 256Fix all the merge conflicts ensuring that the project still builds, by 257periodically running: 258 259`git clean -dfx && make autotools && ./configure && make -j` 260 261Commit the LTP version string 262``` 263git describe <release commit> > VERSION 264git add VERSION 265``` 266 267### Update the Android build targets 268 269Building LTP with the Android build system requires the additional Android 270build configuration files mentioned above. 271A new LTP release may have disabled existing tests or enabled new ones, so the 272Android build configurations must be updated accordingly. 273This is done by the script `android/tools/gen_android_build.sh`: 274 275`git clean -dfx && android/tools/gen_android_build.sh && git clean -dfx && mma .` 276 277This command will possibly update the files `android/Android.ltp.mk`, 278`android/ltp_package_list.mk` and `gen.bp`. 279 280It's a good practice to create an explanatory commit message that presents the 281differences in the test suite. 282`android/tools/compare_ltp_projects.py` is a script that helps comparing the tests available in two different LTP folders. 283 284LTP_NEW=$ANDROID_BUILD_TOP/external/ltp 285LTP_OLD=/tmp/ltp-base 286git archive aosp/master | tar -x -C $LTP_OLD 287android/tools/compare_ltp_projects.py --ltp-new $LTP_NEW --ltp-old $LTP_OLD