xref: /aosp_15_r20/art/test/README.atest.md (revision 795d594fd825385562da6b089ea9b2033f3abf5a)

# Running ART Tests with Atest / Trade Federation

ART Testing has early support for execution in the [Trade
Federation](https://source.android.com/devices/tech/test_infra/tradefed)
("TradeFed") test harness, in particular via the
[Atest](https://source.android.com/compatibility/tests/development/atest)
command line tool.

Atest conveniently takes care of building tests and their dependencies (using
Soong, the Android build system) and executing them using Trade Federation.

See also [README.md](README.md) for a general introduction to ART run-tests and
gtests.

## ART run-tests

### Running ART run-tests on device

ART run-tests are defined in sub-directories of `test/` starting with a number
(e.g. `test/001-HelloWorld`). Each ART run-test is identified in the build
system by a Soong module name following the `art-run-test-`*`<test-directory>`*
format (e.g. `art-run-test-001-HelloWorld`).

You can run a specific ART run-test on device by passing its Soong module name
to Atest:
```bash
atest art-run-test-001-HelloWorld
```

To run all ART run-tests in a single command, the currently recommended way is
to use [test mapping](#test-mapping) (see below).

You can nonetheless run all supported ART run-tests with a single Atest command,
using its support for wildcards:
```bash
atest art-run-test-\*
```

Note: Many ART run-tests are failing with the TradeFed harness as of March 2021,
so the above Atest command will likely report many tests failures. The ART team
is actively working on this issue.

## ART gtests

### Running ART gtests on device

There are three ways to run ART gtests on device:
1. by building "standalone" ART gtests and running them against the active ART
   APEX on the device;
2. by installing the Testing ART APEX (i.e. manually "updating" the ART APEX on
   device); or
3. by setting up a `chroot` environment on the device, and "activating" the
   Testing ART APEX in that environment.

The first approach can be used to test the ART APEX presently residing on a
device (either the original one, located in the "system" partition, or an
updated package, present in the "data" partition).

The second and third approaches make use of the Testing ART APEX
(`com.android.art.testing.apex`), and were previously the only options to run
ART gtests on device, because of build- and link-related limitations (the ART
gtests had to be part of the ART APEX package itself to be able to build and run
properly).

### Running standalone ART gtests on device

Standalone ART gtests are defined as Soong modules `art_standalone_*_tests`. You
can run them individually with Atest, e.g:

```bash
atest art_standalone_cmdline_tests
```

You can also run all of them with a single Atest command, using its support for
wildcards:

```bash
atest art_standalone_\*_tests
```

The previous commands build the corresponding ART gtests and their dependencies,
dynamically link them against local ART APEX libraries (in the source tree), and
run them on device against the active ART APEX.

### Running ART gtests on device by installing the Testing ART APEX

You can run ART gtests on device with the ART APEX installation strategy by
using the following `atest` command:

```bash
atest ArtGtestsTargetInstallApex
```

This command:
1. builds the Testing ART APEX from the Android source tree (including the ART
   gtests);
2. installs the Testing ART APEX using `adb install`;
3. reboots the device;
4. runs the tests; and
5. uninstalls the module.

You can run the tests of a single ART gtest C++ class using the
`ArtGtestsTargetInstallApex:`*`<art-gtest-c++-class>`* syntax, e.g.:
```bash
atest ArtGtestsTargetInstallApex:JniInternalTest
```

This syntax also supports the use of wildcards, e.g.:
```bash
atest ArtGtestsTargetInstallApex:*Test*
```

You can also use Trade Federation options to run a subset of ART gtests, e.g.:
```bash
atest ArtGtestsTargetInstallApex -- \
  --module ArtGtestsTargetInstallApex --test '*JniInternalTest*'
```

You can also pass option `--gtest_filter` to the gtest binary to achieve a
similar effect:
```bash
atest ArtGtestsTargetInstallApex -- \
  --test-arg com.android.tradefed.testtype.GTest:native-test-flag:"--gtest_filter=*JniInternalTest*"
```

### Running ART gtests on device using a `chroot` environment

You can run ART gtests on device with the chroot-based strategy by using the
following command:

```bash
atest ArtGtestsTargetChroot
```

This sequence:
1. builds the Testing ART APEX from the Android source tree (including the ART
   gtests) and all the necessary dependencies for the `chroot` environment;
2. sets up a `chroot` environment on the device;
3. "activates" the Testing ART APEX (and other APEXes that it depends on) in the
   `chroot` environment;
4. runs the tests within the `chroot` environment; and
5. cleans up the environment (deactivates the APEXes and removes the `chroot`
   environment).

### Running ART gtests on host

You first need to build the boot classpath and boot image on host:

```bash
m art-host-tests
```

Then you can use `atest --host` to run host gtests, e.g:

```bash
atest --host art_runtime_tests
```

## Test Mapping

ART Testing supports the execution of tests via [Test
Mapping](https://source.android.com/compatibility/tests/development/test-mapping).
The tests declared in ART's [TEST_MAPPING](../TEST_MAPPING) file are executed
during pre-submit testing (when an ART changelist in Gerrit is verified by
Treehugger) and/or post-submit testing (when a given change is merged in the
Android code base), depending on the "test group" where a test is declared.

### Running tests via Test Mapping with Atest

It is possible to run tests via test mapping locally using Atest.

To run all the tests declared in ART's `TEST_MAPPING` file, use the following
command from the Android source tree top-level directory:
```bash
atest --test-mapping art:all
```
In the previous command, `art` is the (relative) path to the directory
containing the `TEST_MAPPING` file listing the tests to run, while `all` means
that tests declared in all [test
groups](https://source.android.com/compatibility/tests/development/test-mapping#defining_test_groups)
shall be run.

To only run tests executed during pre-submit testing, use:
```bash
atest --test-mapping art:presubmit
```