1## Using CTest with lws 2 3### Updating ancient cmake 4 5You need a recent cmake to have the CTest tests work properly, if you're on an 6older distro you need to update your cmake. Luckily Kitware provide a repo for 7common distros. These instructions work for bionic and xenial. 8 9First remove the old distro cmake and install the pieces needed to get the new repo keys 10 11``` 12# apt purge --auto-remove cmake 13# apt install gnupg wget apt-transport-https ca-certificates 14# wget -O - https://apt.kitware.com/keys/kitware-archive-latest.asc 2>/dev/null | sudo apt-key add - 15# apt edit-sources 16``` 17 18Add the line `deb https://apt.kitware.com/ubuntu/ bionic main` at the end 19replacing `bionic` with `xenial` as needed, and save (:wq). Then 20 21``` 22# apt update 23# apt install cmake 24``` 25 26## Tests live in CMakeLists.txt 27 28The rules for tests are described in ctest / cmake language inside the minimal 29examples and api tests that are enabled by current build options, so you need 30to build with `-DLWS_WITH_MINIMAL_EXAMPLES=1` to build the examples along with 31the library. 32 33The tests are typically running the examples or api tests and regarding the 34process exiting with exit code 0 as success, anything else as failure. 35 36## Generating the tests 37 38The main tests just need `-DLWS_WITH_MINIMAL_EXAMPLES=1`. You can optionally set 39`-DLWS_CTEST_INTERNET_AVAILABLE=0` to indicate you can't run the tests that need 40internet connectivity. 41 42## Preparing to run the tests 43 44The tests have to be able to run without root and without disturbing any other 45install of lws in the build machine. 46 47For that reason you have to do an unprivileged side-install into `../destdir`, 48using `make install DESTDIR=../destdir` from the build directory and perform the 49tests on the pieces in there. 50 51## Running the tests 52 53We must take care to run the pieces (.so etc) we just built, without having 54root access, and not any of the same pieces from some other lws version that may 55have been installed on the build machine. That includes, eg, plugins that 56we just built, to ensure precedence of those in the search path we can set our 57DESTDIR unprivileged install path in `LD_LIBRARY_PATH`. 58 59Then we can run ctest on the unprivileged install. The whole step looks 60something like this: 61 62``` 63build $ make -j12 && \ 64 rm -rf ../destdir && \ 65 make -j12 DESTDIR=../destdir install && \\ 66 LD_LIBRARY_PATH=../destdir/usr/local/share/libwebsockets-test-server/plugins ctest -j2 --output-on-failure 67``` 68 69On windows, it looks like `ctest . -C DEBUG` or RELEASE if that was the build 70type. 71 72Good results look something like this (which tests can run depend on your 73build options) 74 75``` 76Test project /projects/libwebsockets/build 77 Start 71: st_wcs_srv 78 Start 43: st_hcp_srv 79 1/73 Test #71: st_wcs_srv .................................. Passed 5.01 sec 80 Start 19: st_hcmp_srv 81 2/73 Test #43: st_hcp_srv .................................. Passed 5.01 sec 82 Start 17: st_hcm_srv 83 3/73 Test #19: st_hcmp_srv ................................. Passed 5.01 sec 84 Start 55: st_ssproxyctx 85 4/73 Test #17: st_hcm_srv .................................. Passed 5.01 sec 86 Start 52: st_ssproxy 87 5/73 Test #55: st_ssproxyctx ............................... Passed 1.02 sec 88 Start 67: st_sstfproxy 89 6/73 Test #52: st_ssproxy .................................. Passed 1.02 sec 90 Start 60: st_ssprxsmd_sspc 91 7/73 Test #67: st_sstfproxy ................................ Passed 1.01 sec 92 Start 63: st_mulssprxsmd_sspc 93 8/73 Test #60: st_ssprxsmd_sspc ............................ Passed 1.01 sec 94 Start 69: sspc-minimaltf 95 9/73 Test #63: st_mulssprxsmd_sspc ......................... Passed 1.02 sec 96 Start 73: ws-client-spam 9710/73 Test #73: ws-client-spam .............................. Passed 12.21 sec 98 Start 57: sspc-minimaltx 9911/73 Test #57: sspc-minimaltx .............................. Passed 5.90 sec 100 Start 65: mulsspcsmd_sspc 10112/73 Test #65: mulsspcsmd_sspc ............................. Passed 3.58 sec 102 Start 62: sspcsmd_sspc 10313/73 Test #62: sspcsmd_sspc ................................ Passed 1.73 sec 104 Start 22: http-client-multi-h1 10514/73 Test #22: http-client-multi-h1 ........................ Passed 5.04 sec 106 Start 25: http-client-multi-stag 10715/73 Test #25: http-client-multi-stag ...................... Passed 4.53 sec 108 Start 26: http-client-multi-stag-h1 10916/73 Test #26: http-client-multi-stag-h1 ................... Passed 4.40 sec 110 Start 21: http-client-multi 11117/73 Test #21: http-client-multi ........................... Passed 4.37 sec 112 Start 36: http-client-multi-post-h1 11318/73 Test #36: http-client-multi-post-h1 ................... Passed 2.73 sec 114 Start 54: sspc-minimal 11519/73 Test #54: sspc-minimal ................................ Passed 0.93 sec 116 Start 39: http-client-multi-post-stag 11720/73 Test #39: http-client-multi-post-stag ................. Passed 2.29 sec 118 Start 40: http-client-multi-post-stag-h1 11921/73 Test #69: sspc-minimaltf .............................. Passed 49.83 sec 120 Start 35: http-client-multi-post 12122/73 Test #40: http-client-multi-post-stag-h1 .............. Passed 4.30 sec 122 Start 33: http-client-multi-restrict-nopipe-fail 12323/73 Test #35: http-client-multi-post ...................... Passed 3.23 sec 124 Start 28: http-client-multi-stag-h1-pipe 12524/73 Test #33: http-client-multi-restrict-nopipe-fail ...... Passed 2.86 sec 126 Start 32: http-client-multi-restrict-stag-h1-pipe 12725/73 Test #28: http-client-multi-stag-h1-pipe .............. Passed 2.86 sec 128 Start 27: http-client-multi-stag-pipe 12926/73 Test #32: http-client-multi-restrict-stag-h1-pipe ..... Passed 1.51 sec 130 Start 31: http-client-multi-restrict-stag-pipe 13127/73 Test #27: http-client-multi-stag-pipe ................. Passed 1.52 sec 132 Start 34: http-client-multi-restrict-h1-nopipe-fail 13328/73 Test #34: http-client-multi-restrict-h1-nopipe-fail ... Passed 2.78 sec 134 Start 46: http-client-post-m 13529/73 Test #31: http-client-multi-restrict-stag-pipe ........ Passed 2.80 sec 136 Start 42: http-client-multi-post-stag-h1-pipe 13730/73 Test #42: http-client-multi-post-stag-h1-pipe ......... Passed 1.51 sec 138 Start 41: http-client-multi-post-stag-pipe 13931/73 Test #46: http-client-post-m .......................... Passed 1.59 sec 140 Start 48: http-client-post-m-h1 14132/73 Test #48: http-client-post-m-h1 ....................... Passed 1.10 sec 142 Start 23: http-client-multi-pipe 14333/73 Test #41: http-client-multi-post-stag-pipe ............ Passed 1.51 sec 144 Start 29: http-client-multi-restrict-pipe 14534/73 Test #23: http-client-multi-pipe ...................... Passed 1.09 sec 146 Start 24: http-client-multi-h1-pipe 14735/73 Test #29: http-client-multi-restrict-pipe ............. Passed 0.74 sec 148 Start 30: http-client-multi-restrict-h1-pipe 14936/73 Test #24: http-client-multi-h1-pipe ................... Passed 1.14 sec 150 Start 45: http-client-post 15137/73 Test #30: http-client-multi-restrict-h1-pipe .......... Passed 1.14 sec 152 Start 38: http-client-multi-post-h1-pipe 15338/73 Test #45: http-client-post ............................ Passed 0.30 sec 154 Start 37: http-client-multi-post-pipe 15539/73 Test #38: http-client-multi-post-h1-pipe .............. Passed 0.49 sec 156 Start 47: http-client-post-h1 15740/73 Test #37: http-client-multi-post-pipe ................. Passed 0.31 sec 158 Start 50: hs_evlib_foreign_event 15941/73 Test #47: http-client-post-h1 ......................... Passed 0.29 sec 160 Start 66: ss-tf 16142/73 Test #50: hs_evlib_foreign_event ...................... Passed 22.02 sec 162 Start 49: hs_evlib_foreign_uv 16343/73 Test #49: hs_evlib_foreign_uv ......................... Passed 21.03 sec 164 Start 51: ss-warmcat 16544/73 Test #51: ss-warmcat .................................. Passed 2.69 sec 166 Start 59: ss-smd 16745/73 Test #59: ss-smd ...................................... Passed 1.78 sec 168 Start 10: api-test-secure-streams 16946/73 Test #10: api-test-secure-streams ..................... Passed 1.34 sec 170 Start 11: http-client-warmcat 17147/73 Test #11: http-client-warmcat ......................... Passed 0.27 sec 172 Start 58: sspost-warmcat 17348/73 Test #58: sspost-warmcat .............................. Passed 0.84 sec 174 Start 12: http-client-warmcat-h1 17549/73 Test #12: http-client-warmcat-h1 ...................... Passed 0.25 sec 176 Start 2: api-test-jose 17750/73 Test #2: api-test-jose ............................... Passed 0.27 sec 178 Start 70: ws-client-rx-warmcat 17951/73 Test #70: ws-client-rx-warmcat ........................ Passed 0.27 sec 180 Start 56: ki_ssproxyctx 18152/73 Test #56: ki_ssproxyctx ............................... Passed 0.12 sec 182 Start 68: ki_ssproxy 18353/73 Test #68: ki_ssproxy .................................. Passed 0.11 sec 184 Start 64: ki_mulssprxsmd_sspc 18554/73 Test #64: ki_mulssprxsmd_sspc ......................... Passed 0.10 sec 186 Start 61: ki_ssprxsmd_sspc 18755/73 Test #61: ki_ssprxsmd_sspc ............................ Passed 0.11 sec 188 Start 13: http-client-h2-rxflow-warmcat 18956/73 Test #13: http-client-h2-rxflow-warmcat ............... Passed 0.28 sec 190 Start 14: http-client-h2-rxflow-warmcat-h1 19157/73 Test #14: http-client-h2-rxflow-warmcat-h1 ............ Passed 0.34 sec 192 Start 16: http-client-hugeurl-warmcat-h1 19358/73 Test #16: http-client-hugeurl-warmcat-h1 .............. Passed 0.16 sec 194 Start 15: http-client-hugeurl-warmcat 19559/73 Test #15: http-client-hugeurl-warmcat ................. Passed 0.16 sec 196 Start 72: ki_wcs_srv 19760/73 Test #72: ki_wcs_srv .................................. Passed 0.12 sec 198 Start 44: ki_hcp_srv 19961/73 Test #44: ki_hcp_srv .................................. Passed 0.11 sec 200 Start 20: ki_hcmp_srv 20162/73 Test #20: ki_hcmp_srv ................................. Passed 0.11 sec 202 Start 18: ki_hcm_srv 20363/73 Test #18: ki_hcm_srv .................................. Passed 0.11 sec 204 Start 7: api-test-lws_struct_sqlite 20564/73 Test #7: api-test-lws_struct_sqlite .................. Passed 0.03 sec 206 Start 1: api-test-gencrypto 20765/73 Test #1: api-test-gencrypto .......................... Passed 0.02 sec 208 Start 6: api-test-lws_struct-json 20966/73 Test #6: api-test-lws_struct-json .................... Passed 0.01 sec 210 Start 4: api-test-lws_dsh 21167/73 Test #4: api-test-lws_dsh ............................ Passed 0.01 sec 212 Start 8: api-test-lws_tokenize 21368/73 Test #8: api-test-lws_tokenize ....................... Passed 0.01 sec 214 Start 9: api-test-lwsac 21569/73 Test #9: api-test-lwsac .............................. Passed 0.00 sec 216 Start 3: api-test-lejp 21770/73 Test #3: api-test-lejp ............................... Passed 0.00 sec 218 Start 53: ki_ssproxy 21971/73 Test #53: ki_ssproxy .................................. Passed 0.11 sec 22072/73 Test #66: ss-tf ....................................... Passed 55.51 sec 221 Start 5: api-test-lws_smd 22273/73 Test #5: api-test-lws_smd ............................ Passed 4.22 sec 223 224100% tests passed, 0 tests failed out of 73 225 226Total Test time (real) = 137.76 sec 227``` 228 229## Considerations for creating tests 230 231### Timeout 232 233The default test timeout is 1500s, for that reason it's good practice to set 234a more suitable `TIMEOUT` property on every test. 235 236### Working Directory 237 238Server-side test apps usually need to be run from their `./minimal-examples/...` 239directory so they can access their assets like index.html etc. 240 241However when building with `-DLWS_WITH_MBEDTLS=1` then even client-side apps 242need to be run from their directory, since they need to get the trusted CA for 243warmcat.com or libwebsockets.org additionally. 244 245For that reason it's good practice to set the `WORKING_DIRECTORY` property to 246the home dir of the example app in all cases. 247 248### Spawning Buddies 249 250Many networking tests need to either spawn a client or a server in order to 251have a "buddy" to talk to during the test for the opposing side. This is a 252bit awkward in cmake since it does not directly support spawning daemons as 253test dependencies. 254 255Lws provides helper scripts for unix type targets in `./scripts/ctest-background.sh` 256and `./scripts/ctest-background-kill.sh`, which spawn background processes, 257save the pid in a decorated /tmp file and can later take the process down. This 258also has arrangements to dump the log of any background process that exited 259early. 260 261To arrange the buddy to run aligned with the test, you first explain to cmake 262how to start and stop the buddy using phony tests to make a "fixture" in cmake 263terms. 264 265In this example, taken from minimal-http-client-multi, we arrange for 266minimal-http-server-tls to be available for our actual test. The starting and 267stopping definition, for "st_hcm_srv" and "ki_hcm_srv": 268 269``` 270 add_test(NAME st_hcm_srv COMMAND 271 ${CMAKE_SOURCE_DIR}/scripts/ctest-background.sh 272 hcm_srv $<TARGET_FILE:lws-minimal-http-server-tls> 273 --port ${PORT_HCM_SRV} ) 274 add_test(NAME ki_hcm_srv COMMAND 275 ${CMAKE_SOURCE_DIR}/scripts/ctest-background-kill.sh 276 hcm_srv $<TARGET_FILE_NAME:lws-minimal-http-server-tls> 277 --port ${PORT_HCM_SRV}) 278``` 279 280... and binding those together so cmake knows they start and stop a specific 281named fixture "hcm_srv", itself with an 800s timeout 282 283``` 284 set_tests_properties(st_hcm_srv PROPERTIES 285 WORKING_DIRECTORY ${CMAKE_SOURCE_DIR}/minimal-examples/http-server/minimal-http-server-tls 286 FIXTURES_SETUP hcm_srv 287 TIMEOUT 800) 288 set_tests_properties(ki_hcm_srv PROPERTIES 289 FIXTURES_CLEANUP hcm_srv) 290``` 291 292... and finally, adding the "hcm_srv" fixture as a requirement on the actual 293test (http-client-multi) we are testing 294 295``` 296 set_tests_properties(http-client-multi 297 PROPERTIES 298 FIXTURES_REQUIRED "hcm_srv" 299 WORKING_DIRECTORY ${CMAKE_SOURCE_DIR}/minimal-examples/http-client/minimal-http-client-multi 300 TIMEOUT 50) 301``` 302 303Once all that explaining is done, ctest itself will take care about starting 304and killing hcm_srv before and after http-client-multi test. 305 306### Buddy sockets and test concurrency 307 308For tests with local buddies using tcp sockets inside the same VM or systemd- 309nspawn networking context, you cannot just use a well-known port like 7681. 310 311ctest itself is usually executed concurrently, and Sai is typically building 312multiple different instances concurrently as well (typically 3), so it may be 313running different ctests inside the same VM simultaneously. 314 315Different tests can have their own convention for port ranges, to solve the 316problem about Sai running different tests concurrently inside one ctest. 317 318For the case there are multiple ctests running, we can use the env var 319`$ENV{SAI_INSTANCE_IDX}`, which is an ordinal like 0 or 1, to further ensure 320that port selections won't conflict. If not using Sai, you can just set this 321in the evironment yourself to reflect your build instance index. 322 323``` 324 # 325 # instantiate the server per sai builder instance, they are running in the same 326 # machine context in parallel so they can tread on each other otherwise 327 # 328 set(PORT_HCM_SRV "7670") 329 if ("$ENV{SAI_INSTANCE_IDX}" STREQUAL "0") 330 set(PORT_HCM_SRV 7671) 331 endif() 332 if ("$ENV{SAI_INSTANCE_IDX}" STREQUAL "1") 333 set(PORT_HCM_SRV 7672) 334 endif() 335 if ("$ENV{SAI_INSTANCE_IDX}" STREQUAL "2") 336 set(PORT_HCM_SRV 7673) 337 endif() 338 if ("$ENV{SAI_INSTANCE_IDX}" STREQUAL "3") 339 set(PORT_HCM_SRV 7674) 340 endif() 341``` 342 343This is complicated enough that the best approach is copy an existing simple 344case like the CMakeLists.txt for minimal-http-client and change the names and 345ports to be unique. 346