1# CLI tests 2 3The CLI tests are focused on testing the zstd CLI. 4They are intended to be simple tests that the CLI and arguments work as advertised. 5They are not intended to test the library, only the code in `programs/`. 6The library will get incidental coverage, but if you find yourself trying to trigger a specific condition in the library, this is the wrong tool. 7 8## Test runner usage 9 10The test runner `run.py` will run tests against the in-tree build of `zstd` and `datagen` by default. Which means that `zstd` and `datagen` must be built. 11 12The `zstd` binary used can be passed with `--zstd /path/to/zstd`. 13Additionally, to run `zstd` through a tool like `valgrind` or `qemu`, set the `--exec-prefix 'valgrind -q'` flag. 14 15Similarly, the `--datagen`, and `--zstdgrep` flags can be set to specify 16the paths to their respective binaries. However, these tools do not use 17the `EXEC_PREFIX`. 18 19Each test executes in its own scratch directory under `scratch/test/name`. E.g. `scratch/basic/help.sh/`. Normally these directories are removed after the test executes. However, the `--preserve` flag will preserve these directories after execution, and save the tests exit code, stdout, and stderr in the scratch directory to `exit`, `stderr`, and `stdout` respectively. This can be useful for debugging/editing a test and updating the expected output. 20 21### Running all the tests 22 23By default the test runner `run.py` will run all the tests, and report the results. 24 25Examples: 26 27``` 28./run.py 29./run.py --preserve 30./run.py --zstd ../../build/programs/zstd --datagen ../../build/tests/datagen 31``` 32 33### Running specific tests 34 35A set of test names can be passed to the test runner `run.py` to only execute those tests. 36This can be useful for writing or debugging a test, especially with `--preserve`. 37 38The test name can either be the path to the test file, or the test name, which is the path relative to the test directory. 39 40Examples: 41 42``` 43./run.py basic/help.sh 44./run.py --preserve basic/help.sh basic/version.sh 45./run.py --preserve --verbose basic/help.sh 46``` 47 48### Updating exact output 49 50If a test is failing because a `.stderr.exact` or `.stdout.exact` no longer matches, you can re-run the tests with `--set-exact-output` and the correct output will be written. 51 52Example: 53``` 54./run.py --set-exact-output 55./run.py basic/help.sh --set-exact-output 56``` 57 58## Writing a test 59 60Test cases are arbitrary executables, and can be written in any language, but are generally shell scripts. 61After the script executes, the exit code, stderr, and stdout are compared against the expectations. 62 63Each test is run in a clean directory that the test can use for intermediate files. This directory will be cleaned up at the end of the test, unless `--preserve` is passed to the test runner. Additionally, the `setup` script can prepare the directory before the test runs. 64 65### Calling zstd, utilities, and environment variables 66 67The `$PATH` for tests is prepended with the `bin/` sub-directory, which contains helper scripts for ease of testing. 68The `zstd` binary will call the zstd binary specified by `run.py` with the correct `$EXEC_PREFIX`. 69Similarly, `datagen`, `unzstd`, `zstdgrep`, `zstdcat`, etc, are provided. 70 71Helper utilities like `cmp_size`, `println`, and `die` are provided here too. See their scripts for details. 72 73Common shell script libraries are provided under `common/`, with helper variables and functions. They can be sourced with `source "$COMMON/library.sh`. 74 75Lastly, environment variables are provided for testing, which can be listed when calling `run.py` with `--verbose`. 76They are generally used by the helper scripts in `bin/` to coordinate everything. 77 78### Basic test case 79 80When executing your `$TEST` executable, by default the exit code is expected to be `0`. However, you can provide an alternate expected exit code in a `$TEST.exit` file. 81 82When executing your `$TEST` executable, by default the expected stderr and stdout are empty. However, you can override the default by providing one of three files: 83 84* `$TEST.{stdout,stderr}.exact` 85* `$TEST.{stdout,stderr}.glob` 86* `$TEST.{stdout,stderr}.ignore` 87 88If you provide a `.exact` file, the output is expected to exactly match, byte-for-byte. 89 90If you provide a `.glob` file, the output is expected to match the expected file, where each line is interpreted as a glob syntax. Additionally, a line containing only `...` matches all lines until the next expected line matches. 91 92If you provide a `.ignore` file, the output is ignored. 93 94#### Passing examples 95 96All these examples pass. 97 98Exit 1, and change the expectation to be 1. 99 100``` 101exit-1.sh 102--- 103#!/bin/sh 104exit 1 105--- 106 107exit-1.sh.exit 108--- 1091 110--- 111``` 112 113Check the stdout output exactly matches. 114 115``` 116echo.sh 117--- 118#!/bin/sh 119echo "hello world" 120--- 121 122echo.sh.stdout.exact 123--- 124hello world 125--- 126``` 127 128Check the stderr output using a glob. 129 130``` 131random.sh 132--- 133#!/bin/sh 134head -c 10 < /dev/urandom | xxd >&2 135--- 136 137random.sh.stderr.glob 138--- 13900000000: * * * * * * 140``` 141 142Multiple lines can be matched with ... 143 144``` 145random-num-lines.sh 146--- 147#!/bin/sh 148echo hello 149seq 0 $RANDOM 150echo world 151--- 152 153random-num-lines.sh.stdout.glob 154--- 155hello 1560 157... 158world 159--- 160``` 161 162#### Failing examples 163 164Exit code is expected to be 0, but is 1. 165 166``` 167exit-1.sh 168--- 169#!/bin/sh 170exit 1 171--- 172``` 173 174Stdout is expected to be empty, but isn't. 175 176``` 177echo.sh 178--- 179#!/bin/sh 180echo hello world 181``` 182 183Stderr is expected to be hello but is world. 184 185``` 186hello.sh 187--- 188#!/bin/sh 189echo world >&2 190--- 191 192hello.sh.stderr.exact 193--- 194hello 195--- 196``` 197 198### Setup & teardown scripts 199 200Finally, test writing can be eased with setup and teardown scripts. 201Each directory in the test directory is a test-suite consisting of all tests within that directory (but not sub-directories). 202This test suite can come with 4 scripts to help test writing: 203 204* `setup_once` 205* `teardown_once` 206* `setup` 207* `teardown` 208 209The `setup_once` and `teardown_once` are run once before and after all the tests in the suite respectively. 210They operate in the scratch directory for the test suite, which is the parent directory of each scratch directory for each test case. 211They can do work that is shared between tests to improve test efficiency. 212For example, the `dictionaries/setup_once` script builds several dictionaries, for use in the `dictionaries` tests. 213 214The `setup` and `teardown` scripts run before and after each test case respectively, in the test case's scratch directory. 215These scripts can do work that is shared between test cases to make tests more succinct. 216For example, the `dictionaries/setup` script copies the dictionaries built by the `dictionaries/setup_once` script into the test's scratch directory, to make them easier to use, and make sure they aren't accidentally modified. 217 218#### Examples 219 220``` 221basic/setup 222--- 223#!/bin/sh 224# Create some files for testing with 225datagen > file 226datagen > file0 227datagen > file1 228--- 229 230basic/test.sh 231--- 232#!/bin/sh 233zstd file file0 file1 234--- 235 236dictionaries/setup_once 237--- 238#!/bin/sh 239set -e 240 241mkdir files/ dicts/ 242for i in $(seq 10); do 243 datagen -g1000 > files/$i 244done 245 246zstd --train -r files/ -o dicts/0 247--- 248 249dictionaries/setup 250--- 251#!/bin/sh 252 253# Runs in the test case's scratch directory. 254# The test suite's scratch directory that 255# `setup_once` operates in is the parent directory. 256cp -r ../files ../dicts . 257--- 258``` 259