1# Overview 2 3The `prepare_bazel_test_env` script is a proof-of-concept script to create a 4simulated Bazel environment within the Android source tree using targets build 5by the Soong build system. 6 7# Supported Modules 8 9The script currently support generation of a Bazel environment to run the 10following Soong test modules: 11 12* `//platform_testing/tests/example/native:hello_world_test` 13* `//platform_testing/tests/example/jarhosttest: HelloWorldHostTest` 14* `//platform_testing/tests/example/instrumentation: HelloWorldTests` 15 16Additionally, the system supports running the Tradefed console directly through 17the `//tools/tradefederation/core:tradefed` target. 18 19# Usage 20 21There are three actions that the script can perform, which are supplied as the 22first argument to the script: generate, sync, and clean, discussed below. All 23the commands below are written with the `-v` flag for verbose output, however 24this can be safely removed if desired. 25 26## Generate 27 28**Command Line**: `bazel run //build/pesto/experiments/prepare_bazel_test_env -- 29-v generate` 30 31The generate command builds the required targets for a Bazel environment via 32Soong, then stages this environment and associated dependencies at 33`out/pesto-environment/` . 34 35The generate command performs the following actions: 36 371. Builds a set of modules (defined in the packaged templates in the 38 `data/templates` directory) via Soong 392. Creates a prebuilts directory at `out/pesto-environment/prebuilts` which 40 contains symlinks to the Android Build environment provided directories 41 (`ANDROID_HOST_OUT`, `ANDROID_HOST_OUT_TESTCASES`, `ANDROID_PRODUCT_OUT`, 42 `ANDROID_TARGET_OUT_TESTCASES`), and will later be linked to by a 43 `.soong_prebuilts` symlink that is placed adjacent to generated BUILD files. 443. Generates a Bazel environment at `out/pesto-environment/gen` using the 45 packaged environment to determine the locations of files, which are placed 46 in locations relative to the source tree root. For example, the 47 `out/pesto-environment/gen/tools/tradefederation/core/BUILD.bazel` 48 corresponds to a file that will eventually live at 49 `tools/tradefederation/core/BUILD.bazel`. 504. For each BUILD file that is staged, place a `.soong_prebuilts` symlink that 51 links to the aforementioned `prebuilts` directory. 52 53After generation, the environment create can serve as a standalone Bazel 54environment, or can be synced to the source tree using the sync command, 55discussed below. 56 57## Sync 58 59**Command Line**: `bazel run //build/pesto/experiments/prepare_bazel_test_env -- 60-v sync` 61 62The sync command scans the staging directory at `out/pesto-environment/gen` for 63all files and then creates symlinks in the source tree that point at these 64files, additionally each synced BUILD file is provided local access to the 65`prebuilts` directory through a `.soong_prebuilts` symlink. 66 67The sync command performs the following actions: 68 691. Iterates through all files in the staged `out/pesto-environment/gen` 70 directory and create a symlink in the source tree to each file at the proper 71 location, overwriting file in the tree if it exists. For example, the sync 72 action would create a symlink at `packages/modules/adb/BUILD.bazel` that 73 links to `out/pesto-environment/gen/packages/modules/adb/BUILD.bazel`. 742. Create a `.soong_prebuilts` directory in every location in the source tree 753. where a BUILD file is placed, providing local access to the Soong staging 764. directories. 77 78After synchronization, the Bazel environment has been merged with the source 79tree and can be used directly from within the source tree. Additionally, after 80synchronization, subsequent calls to the generate command will propogate 81automatically to the source tree. 82 83## Clean 84 85**Command Line**: `bazel run //build/pesto/experiments/prepare_bazel_test_env -- 86-v clean` 87 88The clean command removes all files that have been created in the tree, and also 89cleans up the environment directory at `out/pesto-environment` . 90 91The clean command performs the following actions: 92 931. For each file packaged with the script, remove the corresponding file from 94 the source tree. 952. For each BUILD file packaged with the script, remove the corresponding 96 `.soong_prebuilts` directory for the source tree. 973. Remove the `out/pesto-environment` directory. 98 99After clean, the environment should be removed from the tree. However, as some 100files may have been overwritten, certain repositories may need to be reset. The 101`build/bazel/rules/BUILD.bazel` file is a notable example that needs to be 102manually reset. 103 104# Adding New Modules. 105 106Adding support for an additional module depends on the type of module to be 107added. Each is discussed below. Of note, all files should be added in the 108`templates` directory and end in the `.template` file extension unless the file 109is a static, non-Bazel file. 110 111## Test Modules (without existing Test Rules) 112 113For targets needing a new test rule, if the test is a Tradefed run test, use the 114existing test rules as a template, otherwise a custom Bazel rule can be added. 115 116An example rule is provided at 117`templates/build/bazel/rules/cc_test.bzl.template` . For a new rule that 118leverages Tradefed, use the above rule as an example of how to package 119dependencies and test artifacts into the runfiles for the test. For each 120Tradefed rule, the required dependencies should be included as private 121attributes, for use by the rule implementation. 122 123``` 124cc_test = rule( 125 _cc_test_impl, 126 attrs = {{ 127 "_adb": attr.label( 128 default = Label("//packages/modules/adb"), 129 allow_single_file = True, 130 ), 131 "_tradefed_launcher": attr.label( 132 default = Label("//tools/tradefederation/core:atest_tradefed"), 133 allow_single_file = True, 134 ), 135 "_tradefed_script_help": attr.label( 136 default = Label("//tools/tradefederation/core:atest_script_help"), 137 ), 138 "_tradefed_jars": attr.label( 139 default = Label("//tools/tradefederation/core:tradefed_lib"), 140 ), 141 "_template": attr.label( 142 default = Label( 143 "//build/bazel/rules:tf_test_executable.sh.template", 144 ), 145 allow_single_file = True, 146 ), 147 "_launcher": attr.label(default = Label("//build/bazel/rules:cc_tf_test_launcher")), 148 "deps": attr.label_list(allow_files = True), 149 }}, 150 executable = True, 151 test = True, 152) 153``` 154 155Additionally, the Soong produced artifacts should also be included as runfiles 156so they can be seen during Tradefed execution. Including a target as runfiles 157here, ensures that the target shows up during execution. Furthermore, it ensures 158that Bazel knows when to rebuild/rerun a test when artifacts change. 159 160``` 161runfiles = ctx.runfiles( 162 files = ctx.files._launcher, 163 transitive_files = depset( 164 transitive = [ 165 depset(ctx.files.deps), 166 depset(ctx.files._adb), 167 depset(ctx.files._tradefed_launcher), 168 depset(ctx.files._tradefed_script_help), 169 depset(ctx.files._tradefed_jars), 170 ], 171 ), 172 ) 173``` 174 175Finally, the test rule should use the tf_test_executable.sh file as its 176executable and provide the proper substitutions to this file, which can be seen 177in the above example rule. The tf_test_executable.sh handles setting important 178variables needed by Tradefed before test execution in a Bazel environment. 179 180``` 181ctx.actions.expand_template( 182 template = ctx.file._template, 183 output = script, 184 substitutions = {{ 185 "{{module_name}}": ctx.label.name, 186 "{{module_path}}": ctx.label.package, 187 "{{tradefed_launcher_module_path}}": ctx.attr._tradefed_launcher.label.package, 188 "{{tradefed_jars_module_path}}": ctx.attr._tradefed_jars.label.package, 189 "{{path_additions}}": ctx.attr._adb.label.package, 190 "{{launcher_path}}": "{{}}/{{}}".format( 191 ctx.attr._launcher.label.package, 192 ctx.attr._launcher.label.name, 193 ), 194 }}, 195 is_executable = True, 196) 197``` 198 199After the rule logic is added, follow the steps in the below section for how to 200add a test target leveraging the newly added rule. 201 202## Test Modules (with existing Test Rules) 203 204For targets where the test rule is already provided (i.e. `cc_test` ), adding a 205new test module requires only adding a new BUILD file (with associated import 206logic). 207 208All added BUILD templates should end in `.template` to ensure Bazel does not see 209these files as part of a package, and should contain the required Soong targets, 210defined like the following: 211 212``` 213# SOONG_TARGET:CtsAppTestCases 214# SOONG_TARGET:org.apache.http.legacy 215``` 216 217Refer to the 218`data/templates/platform_testing/tests/example/native/BUILD.bazel.template` as 219an example of how to import files into the Bazel environment using a genrule. 220The genrule logic in the example template is used to combine files from multiple 221locations into a single target and strip the Soong paths from the files imported 222to Bazel. The below genrule serves as a foundation and copies all files from 223srcs to the files listed in outs. 224 225``` 226genrule(name="hello_world_test_prebuilt", 227 srcs=_LIB_SRCS + _TESTCASE_HOST_SRCS + _TESTCASE_DEVICE_SRCS, 228 outs=_LIB_OUTS + _TESTCASE_HOST_OUTS + _TESTCASE_DEVICE_OUTS, 229 cmd=""" 230 src_files=($(SRCS)) 231 out_files=($(OUTS)) 232 for i in "$${{!src_files[@]}}" 233 do 234 src_file=$${{src_files[$$i]}} 235 out_file=$${{out_files[$$i]}} 236 mkdir -p $$(dirname $$src_file) 237 cp $$src_file $$out_file 238 done 239 """) 240``` 241 242When referring to files imported from Bazel, use the `{prebuilts_dir_name}` 243substitution variable instead of referring to the `.soong_prebuilts` directory 244directly since this may change. 245 246``` 247_LIB_SRCS = glob([ 248 "{prebuilts_dir_name}/host/lib/**/*", 249 "{prebuilts_dir_name}/host/lib64/**/*" 250]) 251``` 252 253Then, the newly imported module can be referenced from an existing test rule as 254a dependency, as is done in the example template. Ensure that the test rule is 255imported, such as in the example file: 256 257``` 258load("//build/bazel/rules:cc_test.bzl", "cc_test") 259. 260. 261. 262cc_test(name="hello_world_test", deps=[":hello_world_test_prebuilt"]) 263``` 264