1.. SPDX-License-Identifier: GPL-2.0-or-later 2 3Writing tests 4============= 5 6This document describes LTP guidelines and it's intended for anybody who wants 7to write or to modify a LTP testcase. It's not a definitive guide and it's not, 8by any means, a substitute for common sense. 9 10Guide to clean and understandable code 11-------------------------------------- 12 13Testcases require that the source code is easy to follow. When a test starts to 14fail, the failure has to be analyzed and clean test codebase makes this task 15much easier and quicker. 16 17Keep things simple 18~~~~~~~~~~~~~~~~~~ 19 20It's worth to keep testcases simple or, better, as simple as possible. 21 22The kernel and libc are tricky beasts and the complexity imposed by their 23interfaces is quite high. Concentrate on the interface you want to test and 24follow the UNIX philosophy. 25 26It's a good idea to make the test as self-contained as possible too, ideally 27tests should not depend on tools or libraries that are not widely available. 28 29Do not reinvent the wheel! 30 31* Use LTP standard interface 32* Do not add custom PASS/FAIL reporting functions 33* Do not write Makefiles from scratch, use LTP build system instead 34* Etc. 35 36Keep functions and variable names short 37~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 38 39Choosing a good name for an API functions or even variables is a difficult 40task do not underestimate it. 41 42There are a couple of customary names for different things that help people to 43understand code, for example: 44 45* For loop variables are usually named with a single letter ``i``, ``j``, ... 46* File descriptors ``fd`` or ``fd_foo``. 47* Number of bytes stored in file are usually named as ``size`` or ``len`` 48* Etc. 49 50Do not over-comment 51~~~~~~~~~~~~~~~~~~~ 52 53Comments can sometimes save your day, but they can easily do more harm than 54good. There has been several cases where comments and actual implementation 55drifted slowly apart which yielded into API misuses and hard to find bugs. 56Remember there is only one thing worse than no documentation: wrong 57documentation. 58 59Ideally, everybody should write code that is obvious, which unfortunately isn't 60always possible. If there is a code that requires to be commented, keep it 61short and to the point. These comments should explain *why* and not *how* 62things are done. 63 64Never ever comment the obvious. 65 66In case of LTP testcases, it's customary to add an asciidoc formatted comment 67paragraph with high-level test description at the beginning of the file right 68under the GPL SPDX header. This helps other people to understand the overall 69goal of the test before they dive into the technical details. It's also 70exported into generated documentation hence it should mostly explain what is 71tested. 72 73DRY (Code duplication) 74~~~~~~~~~~~~~~~~~~~~~~ 75 76Copy & paste is a good servant but very poor master. If you are about to copy a 77large part of the code from one testcase to another, think what would happen if 78you find bug in the code that has been copied all around the tree. What about 79moving it to a library instead? 80 81The same goes for short but complicated parts, whenever you are about to copy & 82paste a syscall wrapper that packs arguments accordingly to machine 83architecture or similarly complicated code, put it into a header instead. 84 85C coding style 86-------------- 87 88LTP adopted `Linux kernel coding style <https://www.kernel.org/doc/html/latest/process/coding-style.html>`_. 89Run ``make check`` in the test's directory and/or use ``make check-$TCID``, it 90uses (among other checks) our vendoring version of 91`checkpatch.pl <https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/plain/scripts/checkpatch.pl>`_ 92script from kernel git tree. 93 94.. note:: 95 If ``make check`` does not report any problems, the code still may be wrong 96 as all tools used for checking only look for common mistakes. 97 98The following linting code can be found when we run ``make check``: 99 100.. list-table:: 101 :header-rows: 1 102 103 * - Linting code 104 - Message 105 - Explanation 106 107 * - LTP-001 108 - Library source files have ``tst_`` prefix 109 - API source code is inside headers in ``include/{empty}*.h``, 110 ``include/lapi/{empty}*.h`` (backward compatibility for old kernel and 111 libc) and C sources in ``lib/{empty}*.c``. Files must have ``tst_`` 112 prefix. 113 114 * - LTP-002 115 - ``TST_RET`` and ``TST_ERR`` are never modified by test library functions 116 - The test author is guaranteed that the test API will not modify these 117 variables. This prevents silent errors where the return value and 118 errno are overwritten before the test has chance to check them. 119 120 The macros which are clearly intended to update these variables. That 121 is ``TEST`` and those in ``tst_test_macros.h``. Are of course allowed to 122 update these variables. 123 124 * - LTP-003 125 - Externally visible library symbols have the ``tst_`` prefix 126 - Functions, types and variables in the public test API should have the 127 ``tst_`` prefix. With some exceptions for symbols already prefixed with 128 ``safe_`` or ``ltp_``. 129 130 Static (private) symbols should not have the prefix. 131 132 * - LTP-004 133 - Test executable symbols are marked ``static`` 134 - Test executables should not export symbols unnecessarily. This means 135 that all top-level variables and functions should be marked with the 136 ``static`` keyword. The only visible symbols should be those included 137 from shared object files. 138 139 * - LTP-005 140 - Array must terminate with a sentinel value (i.e. ``NULL`` or ``{}``) 141 - When defining arrays in the ``struct tst_test`` structure, we need to 142 end the array items with a sentinel ``NULL`` value. 143 144Shell coding style 145------------------ 146 147When writing testcases in shell, write in *portable shell* only, it's a good 148idea to try to run the test using alternative shell (alternative to bash, for 149example dash) too. 150 151*Portable shell* means Shell Command Language as defined by POSIX with an 152exception of few widely used extensions, namely **local** keyword used inside of 153functions and ``-o`` and ``-a`` test parameters (that are marked as obsolete in 154POSIX). 155 156You can either try to run the testcases in Debian which has ``/bin/sh`` pointing 157to ``dash`` by default, or to install ``dash`` on your favorite distribution, 158then use it to run tests. If your distribution lacks ``dash`` package, you can 159always compile it from `sources <http://gondor.apana.org.au/~herbert/dash/files/>`_. 160 161Run ``make check`` in the test's directory and/or use ``make check-$TCID.sh``. 162It uses (among other checks) our vendoring version of 163`checkbashism.pl <https://salsa.debian.org/debian/devscripts/raw/master/scripts/checkbashisms.pl>`_ 164from Debian that is used to check for non-portable shell code. 165 166.. note:: 167 168 If ``make check`` does not report any problems the code still may be wrong, 169 as ``checkbashisms.pl`` is used for checking only common mistakes. 170 171Here there are some common sense style rules for shell 172 173* Keep lines under 80 chars 174* Use tabs for indentation 175* Keep things simple, avoid unnecessary subshells 176* Don't do confusing things (i.e. don't name your functions like common shell 177 commands, etc.) 178* Quote variables 179* Be consistent 180 1813 Backwards compatibility 182~~~~~~~~~~~~~~~~~~~~~~~~~ 183 184LTP test should be as backward compatible as possible. Think of an enterprise 185distributions with long term support (more than five years since the initial 186release) or of an embedded platform that needs to use several years old 187toolchain supplied by the manufacturer. 188 189Therefore LTP test for more current features should be able to cope with older 190systems. It should at least compile fine and if it's not appropriate for the 191configuration it should return ``TCONF``. 192 193There are several types of checks we use: 194 195* The *configure script* is usually used to detect availability of a function 196 declarations in system headers. It's used to disable tests at compile time or 197 to enable fallback definitions. 198 199* Checking the ``errno`` value is another type of runtime check. Most of the 200 syscalls returns either ``EINVAL`` or ``ENOSYS`` when syscall was not 201 implemented or was disabled upon kernel compilation. 202 203* LTP has kernel version detection that can be used to disable tests at runtime. 204 Unfortunately, the kernel version does not always corresponds to a well 205 defined feature set, as distributions tend to backport hundreds of patches 206 while the kernel version stays the same. Use with caution. 207 208* Lately, we added a kernel ``.config`` parser. A test can define a boolean 209 expression of kernel config variables that has to be satisfied in order to run 210 a test. At the moment, this is mostly used for kernel namespaces. 211 212* Sometimes it makes sense to define a few macros instead of creating a 213 configure test. One example is Linux specific POSIX clock ids in 214 :master:`include/lapi/posix_clocks.h`. 215 216Dealing with messed up legacy code 217~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 218 219LTP still contains a lot of old and messy code and we are cleaning it up as 220fast as we can but, despite the decade of efforts, there is still a lot of it. 221If you start modifying old or a messy testcase and your changes are more 222complicated than simple typo fixes, you should convert the test into a new 223library first. 224 225It's also much easier to review the changes if you split them into a smaller 226logical groups. The same goes for moving files: if you need to rename or to move 227files, do it in a separate patch. 228 229License 230~~~~~~~ 231 232Code contributed to LTP should be licensed under GPLv2+ (GNU GPL version 2 or 233any later version). 234 235Use ``SPDX-License-Identifier: GPL-2.0-or-later`` 236 237LTP Structure 238------------- 239 240The structure of LTP is quite simple. Each test is a binary written either in 241portable shell or C. The test gets a configuration via environment variables 242and/or command line parameters, it prints additional information into the 243stdout and reports overall success/failure via the exit value. 244 245Tests are generally placed under the :master:`testcases` directory. Everything that 246is a syscall or (slightly confusingly) libc syscall wrapper, goes under 247:master:`testcases/kernel/syscalls/`. 248 249There is also :master:`testcases/open_posix_testsuite/` which is a well maintained 250fork of the Open POSIX testsuite project, that has been dead since 2005. 251 252We also have a number of directories with tests for more specific features, such 253as containers, etc. 254 255Runtest Files 256~~~~~~~~~~~~~ 257 258The list of tests to be executed is stored in runtest files under the 259:master:`runtest` directory. The default set of runtest files to be executed is 260stored in :master:`scenario_groups/default`. When you add a test, you should add 261corresponding entries into some runtest file(s) as well. 262 263Each line of runtest file contains one test. The first item is the test name. 264All other items, separated by space will be executed as a command. 265 266.. code-block:: bash 267 268 shell_test01 echo "SUCCESS" | shell_pipe01.sh 269 splice02 splice02 -s 20 270 271Blank lines and lines starting with a ``#`` (comments) are ignored. 272 273Syscalls tests, placed under :master:`testcases/kernel/syscalls/`, use 274:master:`runtest/syscalls` file. For kernel related tests for memory management we 275have :master:`runtest/mm`, etc. 276 277.. note:: 278 279 runtest files should have one entry per a test. Creating a 280 wrapper that runs all your tests and adding it as a single test 281 into runtest file is strongly discouraged. 282 283Datafiles 284--------- 285 286If your test needs data files, these should be put into a subdirectory 287named ``datafiles`` and installed into the ``testcases/data/$TCID`` directory. 288This will require to add ``INSTALL_DIR := testcases/data/TCID`` into 289correspondent ``datafiles/Makefile``. 290 291You can obtain path to datafiles via ``$TST_DATAROOT`` provided by ``test.sh`` 292or via C function ``tst_dataroot()`` provided by libltp: 293 294.. code-block:: c 295 296 const char *dataroot = tst_dataroot(); 297 298Datafiles can also be accessed as ``$LTPROOT/testcases/data/$TCID/...``, 299but ``$TST_DATAROOT`` and ``tst_dataroot()`` are preferred, as these can be used 300when running testcases directly in git tree as well as from install location. 301 302Sub-executables 303~~~~~~~~~~~~~~~ 304 305If your test needs to execute a binary, place it in the same directory of the 306testcase and name the binary with ``$TESTNAME_`` prefix, where ``$TESTNAME`` is 307the name of the test binary. Once the test is executed by the framework, the 308path to the directory with all LTP binaries is added to the ``$PATH`` and you 309can execute it via its name. 310 311.. note:: 312 313 If you need to execute a test from the LTP tree, you can add ``PATH`` to 314 the current directory with ``PATH="$PATH:$PWD" ./foo01``. 315 316Test Contribution Checklist 317--------------------------- 318 319#. Test compiles and it runs fine (check with ``-i 10`` and ``-i 0`` too) 320#. ``make check`` should not emit any warnings for the test you are working on 321 (hint: run it in the test's directory and/or use ``make check-$TCID``) 322#. The runtest entries are in place 323#. New test binaries are added into the corresponding ``.gitignore`` files 324#. Patches apply over the latest git 325 326About .gitignore files 327~~~~~~~~~~~~~~~~~~~~~~ 328 329There are numerous ``.gitignore`` files in the LTP tree. Usually, there is a 330``.gitignore`` file for a group of tests. The reason of this setup is simple: 331it's easier to maintain a ``.gitignore`` file per tests' directory, rather 332than having a single file in the project root directory. In this way, we don't 333have to update all the gitignore files when moving directories, and they get 334deleted automatically when a directory with tests is removed. 335 336Testing pre-release kernel features 337----------------------------------- 338 339Tests for features not yet in the mainline kernel release are accepted. However, 340they must be added only to :master:`runtest/staging`. Once a feature is part 341of the stable kernel ABI, the associated test must be moved out of staging. 342 343Testing builds with GitHub Actions 344---------------------------------- 345 346Master branch is tested in GitHub :repo:`actions` 347to ensure LTP builds in various distributions, including old, current and 348bleeding edge. ``gcc`` and ``clang`` toolchains are also tested for various 349architectures using cross-compilation. For a full list of tested distros, please 350check :master:`.github/workflows/ci-docker-build.yml`. 351 352.. note:: 353 354 Passing the GitHub Actions CI means that LTP compiles in a variety of 355 different distributions on their **newest releases**. 356 The CI also checks for code linting, running ``make check`` in the whole 357 LTP project. 358 359LTP C And Shell Test API Comparison 360----------------------------------- 361 362.. list-table:: 363 :header-rows: 1 364 365 * - C API ``struct tst_test`` members 366 - Shell API ``$TST_*`` variables 367 368 * - .all_filesystems 369 - TST_ALL_FILESYSTEMS 370 371 * - .bufs 372 - \- 373 374 * - .caps 375 - \- 376 377 * - .child_needs_reinit 378 - not applicable 379 380 * - .cleanup 381 - TST_CLEANUP 382 383 * - .dev_extra_opts 384 - TST_DEV_EXTRA_OPTS 385 386 * - .dev_fs_opts 387 - TST_DEV_FS_OPTS 388 389 * - .dev_fs_type 390 - TST_FS_TYPE 391 392 * - .dev_min_size 393 - not applicable 394 395 * - .format_device 396 - TST_FORMAT_DEVICE 397 398 * - .max_runtime 399 - \- 400 401 * - .min_cpus 402 - not applicable 403 404 * - .min_kver 405 - TST_MIN_KVER 406 407 * - .min_mem_avail 408 - not applicable 409 410 * - .mnt_flags 411 - TST_MNT_PARAMS 412 413 * - .min_swap_avail 414 - not applicable 415 416 * - .mntpoint | .mnt_data 417 - TST_MNTPOINT 418 419 * - .mount_device 420 - TST_MOUNT_DEVICE 421 422 * - .needs_cgroup_ctrls 423 - \- 424 425 * - .needs_checkpoints 426 - TST_NEEDS_CHECKPOINTS 427 428 * - .needs_cmds 429 - TST_NEEDS_CMDS 430 431 * - .needs_devfs 432 - \- 433 434 * - .needs_device 435 - TST_NEEDS_DEVICE 436 437 * - .needs_drivers 438 - TST_NEEDS_DRIVERS 439 440 * - .needs_kconfigs 441 - TST_NEEDS_KCONFIGS 442 443 * - .needs_overlay 444 - \- 445 446 * - .needs_rofs 447 - \- 448 449 * - .needs_root 450 - TST_NEEDS_ROOT 451 452 * - .needs_tmpdir 453 - TST_NEEDS_TMPDIR 454 455 * - .options 456 - TST_PARSE_ARGS | TST_OPTS 457 458 * - .resource_files 459 - \- 460 461 * - .restore_wallclock 462 - not applicable 463 464 * - .sample 465 - \- 466 467 * - .save_restore 468 - \- 469 470 * - .scall 471 - not applicable 472 473 * - .setup 474 - TST_SETUP 475 476 * - .skip_filesystems 477 - TST_SKIP_FILESYSTEMS 478 479 * - .skip_in_compat 480 - \- 481 482 * - .skip_in_lockdown 483 - TST_SKIP_IN_LOCKDOWN 484 485 * - .skip_in_secureboot 486 - TST_SKIP_IN_SECUREBOOT 487 488 * - .supported_archs 489 - not applicable 490 491 * - .tags 492 - \- 493 494 * - .taint_check 495 - \- 496 497 * - .tcnt 498 - TST_CNT 499 500 * - .tconf_msg 501 - not applicable 502 503 * - .test | .test_all 504 - TST_TESTFUNC 505 506 * - .test_variants 507 - \- 508 509 * - .timeout 510 - TST_TIMEOUT 511 512 * - .tst_hugepage 513 - not applicable 514 515 * - .ulimit 516 - not applicable 517 518 * - not applicable 519 - TST_NEEDS_KCONFIGS_IFS 520 521 * - not applicable 522 - TST_NEEDS_MODULE 523 524 * - not applicable 525 - TST_POS_ARGS 526 527 * - not applicable 528 - TST_USAGE 529 530.. list-table:: 531 :header-rows: 1 532 533 * - C API other structs 534 - Shell API ``$TST_*`` variables 535 536 * - struct tst_device 537 - TST_DEVICE 538