1# Porting to Other Implementations 2 3## Introduction 4 5This document provides an overview of the test runner and how to 6integrate it with other stacks. So far we have it working with 7BoringSSL and some incomplete integrations with NSS and OpenSSL. 8 9Note that supporting non-BoringSSL implementations is a work in 10progress and interfaces may change in the future. Consumers should pin 11to a particular revision rather than using BoringSSL’s master branch 12directly. As we gain experience with other implementations, we hope to 13make further improvements to portability, so please contact 14[email protected] and [email protected] if implementing a new shim. 15 16 17## Integration Architecture 18 19The test runner integrates with the TLS stack under test through a 20“shim”: a command line program which encapsulates the stack. By 21default, the shim points to the BoringSSL shim in the same source 22tree, but any program can be supplied via the `-shim-path` flag. The 23runner opens up a server socket and provides the shim with `-port`, `-shim-id` 24and optional `-ipv6` arguments. 25 26For each connection, the shim should connect to loopback as a TCP client on 27the specified port, using IPv6 if `-ipv6` is specified and IPv4 otherwise. 28It then sends the shim ID as a 64-bit, little-endian integer and proceeds with 29the test. The shim is a TCP client even when testing DTLS or TLS server 30behavior. For DTLS, there is a small framing layer that gives packet boundaries 31over TCP. The shim can also pass a variety of command line arguments 32which are used to configure the stack under test. These can be found at 33`test_config.cc`. 34 35The shim reports success by exiting with a `0` error code and failure by 36reporting a non-zero error code and generally sending a textual error 37value to stderr. Many of the tests expect specific error string (such 38as `NO_SHARED_CIPHER`) that indicates what went wrong. 39 40 41## Compatibility Issues 42 43There are a number of situations in which the runner might succeed 44with some tests and not others: 45 46* Defects in the stack under test 47* Features which haven’t yet been implemented 48* Failure to implement one or more of the command line flags the runner uses with the shim 49* Disagreement about the right behavior/interpretation of the spec 50 51 52We have implemented several features which allow implementations to ease these compatibility issues. 53 54### Configuration File 55 56The runner can be supplied with a JSON configuration file which is 57intended to allow for a per-stack mapping. This file currently takes 58two directives: 59 60 61* `DisabledTests`: A JSON map consisting of the pattern matching the 62 tests to be disabled as the key and some sort of reason why it was 63 disabled as the value. The key is used as a match against the test 64 name. The value is ignored and is just used for documentation 65 purposes so you can remember why you disabled a 66 test. `-include-disabled` overrides this filter. 67 68* `ErrorMap`: A JSON map from the internal errors the runner expects to 69 the error strings that your implementation spits out. Generally 70 you’ll need to map every error, but if you also provide the 71 ` -loose-errors` flag, then every un-mapped error just gets mapped to 72 the empty string and treated as if it matched every error the runner 73 expects. 74 75 76The `-shim-config` flag is used to provide the config file. 77 78 79### Unimplemented Features 80If the shim encounters some request from the runner that it knows it 81can’t fulfill (e.g., a command line flag that it doesn’t recognize), 82then it can exit with the special code `89`. Shims are recommended to 83use this exit code on unknown command-line arguments. 84 85The test runner interprets this as “unimplemented” and skips the 86test. If run normally, this will cause the test runner to report that 87the entire test suite failed. The `-allow-unimplemented` flag suppresses 88this behavior and causes the test runner to ignore these tests for the 89purpose of evaluating the success or failure of the test suite. 90 91 92### Malloc Tests 93 94The test runner can also be used to stress malloc failure 95codepaths. If passed `-malloc-test=0`, the runner will run each test 96repeatedly with an incrementing `MALLOC_NUMBER_TO_FAIL` environment 97variable. The shim should then replace the malloc implementation with 98one which fails at the specified number of calls. If there are not 99enough calls to reach the number, the shim should fail with exit code 100`88`. This signals to the runner that the test has completed. 101 102Historically, BoringSSL did this by replacing the actual `malloc` 103symbol, but we have found hooking the library's `malloc` wrapper, under a 104test-only build configuration, to be more straightforward. See `crypto/mem.c` 105for an example which handles the environment variables in `OPENSSL_malloc`. 106 107Note these tests are slow and will hit Go's test timeout. Pass `-timeout 72h` to 108avoid crashing after 10 minutes. 109 110 111## Example: Running Against NSS 112 113``` 114DYLD_LIBRARY_PATH=~/dev/nss-dev/nss-sandbox/dist/Darwin15.6.0_64_DBG.OBJ/lib go test -shim-path ~/dev/nss-dev/nss-sandbox/dist/Darwin15.6.0_64_DBG.OBJ/bin/nss_bogo_shim -loose-errors -allow-unimplemented -shim-config ~/dev/nss-dev/nss-sandbox/nss/external_tests/nss_bogo_shim/config.json 115``` 116