1ctest_test
2----------
3
4Perform the :ref:`CTest Test Step` as a :ref:`Dashboard Client`.
5
6::
7
8  ctest_test([BUILD <build-dir>] [APPEND]
9             [START <start-number>]
10             [END <end-number>]
11             [STRIDE <stride-number>]
12             [EXCLUDE <exclude-regex>]
13             [INCLUDE <include-regex>]
14             [EXCLUDE_LABEL <label-exclude-regex>]
15             [INCLUDE_LABEL <label-include-regex>]
16             [EXCLUDE_FIXTURE <regex>]
17             [EXCLUDE_FIXTURE_SETUP <regex>]
18             [EXCLUDE_FIXTURE_CLEANUP <regex>]
19             [PARALLEL_LEVEL <level>]
20             [RESOURCE_SPEC_FILE <file>]
21             [TEST_LOAD <threshold>]
22             [SCHEDULE_RANDOM <ON|OFF>]
23             [STOP_ON_FAILURE]
24             [STOP_TIME <time-of-day>]
25             [RETURN_VALUE <result-var>]
26             [CAPTURE_CMAKE_ERROR <result-var>]
27             [REPEAT <mode>:<n>]
28             [OUTPUT_JUNIT <file>]
29             [QUIET]
30             )
31
32..
33   _note: If updating the argument list here, please also update the argument
34   list documentation for :command:`ctest_memcheck` as well.
35
36Run tests in the project build tree and store results in
37``Test.xml`` for submission with the :command:`ctest_submit` command.
38
39The options are:
40
41``BUILD <build-dir>``
42  Specify the top-level build directory.  If not given, the
43  :variable:`CTEST_BINARY_DIRECTORY` variable is used.
44
45``APPEND``
46  Mark ``Test.xml`` for append to results previously submitted to a
47  dashboard server since the last :command:`ctest_start` call.
48  Append semantics are defined by the dashboard server in use.
49  This does *not* cause results to be appended to a ``.xml`` file
50  produced by a previous call to this command.
51
52``START <start-number>``
53  Specify the beginning of a range of test numbers.
54
55``END <end-number>``
56  Specify the end of a range of test numbers.
57
58``STRIDE <stride-number>``
59  Specify the stride by which to step across a range of test numbers.
60
61``EXCLUDE <exclude-regex>``
62  Specify a regular expression matching test names to exclude.
63
64``INCLUDE <include-regex>``
65  Specify a regular expression matching test names to include.
66  Tests not matching this expression are excluded.
67
68``EXCLUDE_LABEL <label-exclude-regex>``
69  Specify a regular expression matching test labels to exclude.
70
71``INCLUDE_LABEL <label-include-regex>``
72  Specify a regular expression matching test labels to include.
73  Tests not matching this expression are excluded.
74
75``EXCLUDE_FIXTURE <regex>``
76  .. versionadded:: 3.7
77
78  If a test in the set of tests to be executed requires a particular fixture,
79  that fixture's setup and cleanup tests would normally be added to the test
80  set automatically. This option prevents adding setup or cleanup tests for
81  fixtures matching the ``<regex>``. Note that all other fixture behavior is
82  retained, including test dependencies and skipping tests that have fixture
83  setup tests that fail.
84
85``EXCLUDE_FIXTURE_SETUP <regex>``
86  .. versionadded:: 3.7
87
88  Same as ``EXCLUDE_FIXTURE`` except only matching setup tests are excluded.
89
90``EXCLUDE_FIXTURE_CLEANUP <regex>``
91  .. versionadded:: 3.7
92
93  Same as ``EXCLUDE_FIXTURE`` except only matching cleanup tests are excluded.
94
95``PARALLEL_LEVEL <level>``
96  Specify a positive number representing the number of tests to
97  be run in parallel.
98
99``RESOURCE_SPEC_FILE <file>``
100  .. versionadded:: 3.16
101
102  Specify a
103  :ref:`resource specification file <ctest-resource-specification-file>`. See
104  :ref:`ctest-resource-allocation` for more information.
105
106``TEST_LOAD <threshold>``
107  .. versionadded:: 3.4
108
109  While running tests in parallel, try not to start tests when they
110  may cause the CPU load to pass above a given threshold.  If not
111  specified the :variable:`CTEST_TEST_LOAD` variable will be checked,
112  and then the ``--test-load`` command-line argument to :manual:`ctest(1)`.
113  See also the ``TestLoad`` setting in the :ref:`CTest Test Step`.
114
115``REPEAT <mode>:<n>``
116  .. versionadded:: 3.17
117
118  Run tests repeatedly based on the given ``<mode>`` up to ``<n>`` times.
119  The modes are:
120
121  ``UNTIL_FAIL``
122    Require each test to run ``<n>`` times without failing in order to pass.
123    This is useful in finding sporadic failures in test cases.
124
125  ``UNTIL_PASS``
126    Allow each test to run up to ``<n>`` times in order to pass.
127    Repeats tests if they fail for any reason.
128    This is useful in tolerating sporadic failures in test cases.
129
130  ``AFTER_TIMEOUT``
131    Allow each test to run up to ``<n>`` times in order to pass.
132    Repeats tests only if they timeout.
133    This is useful in tolerating sporadic timeouts in test cases
134    on busy machines.
135
136``SCHEDULE_RANDOM <ON|OFF>``
137  Launch tests in a random order.  This may be useful for detecting
138  implicit test dependencies.
139
140``STOP_ON_FAILURE``
141  .. versionadded:: 3.18
142
143  Stop the execution of the tests once one has failed.
144
145``STOP_TIME <time-of-day>``
146  Specify a time of day at which the tests should all stop running.
147
148``RETURN_VALUE <result-var>``
149  Store in the ``<result-var>`` variable ``0`` if all tests passed.
150  Store non-zero if anything went wrong.
151
152``CAPTURE_CMAKE_ERROR <result-var>``
153  .. versionadded:: 3.7
154
155  Store in the ``<result-var>`` variable -1 if there are any errors running
156  the command and prevent ctest from returning non-zero if an error occurs.
157
158``OUTPUT_JUNIT <file>``
159  .. versionadded:: 3.21
160
161  Write test results to ``<file>`` in JUnit XML format. If ``<file>`` is a
162  relative path, it will be placed in the build directory. If ``<file>``
163  already exists, it will be overwritten. Note that the resulting JUnit XML
164  file is **not** uploaded to CDash because it would be redundant with
165  CTest's ``Test.xml`` file.
166
167``QUIET``
168  .. versionadded:: 3.3
169
170  Suppress any CTest-specific non-error messages that would have otherwise
171  been printed to the console.  Output from the underlying test command is not
172  affected.  Summary info detailing the percentage of passing tests is also
173  unaffected by the ``QUIET`` option.
174
175See also the :variable:`CTEST_CUSTOM_MAXIMUM_PASSED_TEST_OUTPUT_SIZE`
176and :variable:`CTEST_CUSTOM_MAXIMUM_FAILED_TEST_OUTPUT_SIZE` variables.
177
178.. _`Additional Test Measurements`:
179
180Additional Test Measurements
181^^^^^^^^^^^^^^^^^^^^^^^^^^^^
182
183CTest can parse the output of your tests for extra measurements to report
184to CDash.
185
186When run as a :ref:`Dashboard Client`, CTest will include these custom
187measurements in the ``Test.xml`` file that gets uploaded to CDash.
188
189Check the `CDash test measurement documentation
190<https://github.com/Kitware/CDash/blob/master/docs/test_measurements.md>`_
191for more information on the types of test measurements that CDash recognizes.
192
193.. versionadded: 3.22
194  CTest can parse custom measurements from tags named
195  ``<CTestMeasurement>`` or ``<CTestMeasurementFile>``. The older names
196  ``<DartMeasurement>`` and ``<DartMeasurementFile>`` are still supported.
197
198The following example demonstrates how to output a variety of custom test
199measurements.
200
201.. code-block:: c++
202
203   std::cout <<
204     "<CTestMeasurement type=\"numeric/double\" name=\"score\">28.3</CTestMeasurement>"
205     << std::endl;
206
207   std::cout <<
208     "<CTestMeasurement type=\"text/string\" name=\"color\">red</CTestMeasurement>"
209     << std::endl;
210
211   std::cout <<
212     "<CTestMeasurement type=\"text/link\" name=\"CMake URL\">https://cmake.org</CTestMeasurement>"
213     << std::endl;
214
215   std::cout <<
216     "<CTestMeasurement type=\"text/preformatted\" name=\"Console Output\">" <<
217     "line 1.\n" <<
218     "  \033[31;1m line 2. Bold red, and indented!\033[0;0ml\n" <<
219     "line 3. Not bold or indented...\n" <<
220     "</CTestMeasurement>" << std::endl;
221
222Image Measurements
223""""""""""""""""""
224
225The following example demonstrates how to upload test images to CDash.
226
227.. code-block:: c++
228
229   std::cout <<
230     "<CTestMeasurementFile type=\"image/jpg\" name=\"TestImage\">" <<
231     "/dir/to/test_img.jpg</CTestMeasurementFile>" << std::endl;
232
233   std::cout <<
234     "<CTestMeasurementFile type=\"image/gif\" name=\"ValidImage\">" <<
235     "/dir/to/valid_img.gif</CTestMeasurementFile>" << std::endl;
236
237   std::cout <<
238     "<CTestMeasurementFile type=\"image/png\" name=\"AlgoResult\"> <<
239     "/dir/to/img.png</CTestMeasurementFile>"
240     << std::endl;
241
242Images will be displayed together in an interactive comparison mode on CDash
243if they are provided with two or more of the following names.
244
245* ``TestImage``
246* ``ValidImage``
247* ``BaselineImage``
248* ``DifferenceImage2``
249
250By convention, ``TestImage`` is the image generated by your test, and
251``ValidImage`` (or ``BaselineImage``) is basis of comparison used to determine
252if the test passed or failed.
253
254If another image name is used it will be displayed by CDash as a static image
255separate from the interactive comparison UI.
256
257Attached Files
258""""""""""""""
259
260.. versionadded:: 3.21
261
262The following example demonstrates how to upload non-image files to CDash.
263
264.. code-block:: c++
265
266   std::cout <<
267     "<CTestMeasurementFile type=\"file\" name=\"TestInputData1\">" <<
268     "/dir/to/data1.csv</CTestMeasurementFile>\n"                   <<
269     "<CTestMeasurementFile type=\"file\" name=\"TestInputData2\">" <<
270     "/dir/to/data2.csv</CTestMeasurementFile>"                     << std::endl;
271
272If the name of the file to upload is known at configure time, you can use the
273:prop_test:`ATTACHED_FILES` or :prop_test:`ATTACHED_FILES_ON_FAIL` test
274properties instead.
275
276Custom Details
277""""""""""""""
278
279.. versionadded:: 3.21
280
281The following example demonstrates how to specify a custom value for the
282``Test Details`` field displayed on CDash.
283
284.. code-block:: c++
285
286   std::cout <<
287     "<CTestDetails>My Custom Details Value</CTestDetails>" << std::endl;
288
289.. _`Additional Labels`:
290
291Additional Labels
292"""""""""""""""""
293
294.. versionadded:: 3.22
295
296The following example demonstrates how to add additional labels to a test
297at runtime.
298
299.. code-block:: c++
300
301   std::cout <<
302     "<CTestLabel>Custom Label 1</CTestLabel>\n" <<
303     "<CTestLabel>Custom Label 2</CTestLabel>"   << std::endl;
304
305Use the :prop_test:`LABELS` test property instead for labels that can be
306determined at configure time.
307