xref: /aosp_15_r20/external/json-schema-validator/src/test/suite/README.md (revision 78c4dd6aa35290980cdcd1623a7e337e8d021c7c) 
1# JSON Schema Test Suite
2
3[![Contributor Covenant](https://img.shields.io/badge/Contributor%20Covenant-2.1-4baaaa.svg)](https://github.com/json-schema-org/.github/blob/main/CODE_OF_CONDUCT.md)
4[![Project Status: Active – The project has reached a stable, usable state and is being actively developed.](https://www.repostatus.org/badges/latest/active.svg)](https://www.repostatus.org/#active)
5[![Financial Contributors on Open Collective](https://opencollective.com/json-schema/all/badge.svg?label=financial+contributors)](https://opencollective.com/json-schema)
6
7[![DOI](https://zenodo.org/badge/5952934.svg)](https://zenodo.org/badge/latestdoi/5952934)
8[![Build Status](https://github.com/json-schema-org/JSON-Schema-Test-Suite/workflows/Test%20Suite%20Sanity%20Checking/badge.svg)](https://github.com/json-schema-org/JSON-Schema-Test-Suite/actions?query=workflow%3A%22Test+Suite+Sanity+Checking%22)
9
10This repository contains a set of JSON objects that implementers of JSON Schema validation libraries can use to test their validators.
11
12It is meant to be language agnostic and should require only a JSON parser.
13The conversion of the JSON objects into tests within a specific language and test framework of choice is left to be done by the validator implementer.
14
15## Coverage
16
17All JSON Schema specification releases should be well covered by this suite, including drafts 2020-12, 2019-09, 07, 06, 04 and 03.
18Drafts 04 and 03 are considered "frozen" in that less effort is put in to backport new tests to these versions.
19
20Additional coverage is always welcome, particularly for bugs encountered in real-world implementations.
21If you see anything missing or incorrect, please feel free to [file an issue](https://github.com/json-schema-org/JSON-Schema-Test-Suite/issues) or [submit a PR](https://github.com/json-schema-org/JSON-Schema-Test-Suite).
22
23@gregsdennis has also started a separate [test suite](https://github.com/gregsdennis/json-schema-vocab-test-suites) that is modelled after this suite to cover third-party vocabularies.
24
25## Introduction to the Test Suite Structure
26
27The tests in this suite are contained in the `tests` directory at the root of this repository.
28Inside that directory is a subdirectory for each released version of the specification.
29
30The structure and contents of each file in these directories is described below.
31
32In addition to the version-specific subdirectories, two additional directories are present:
33
341. `draft-next/`: containing tests for the next version of the specification whilst it is in development
352. `latest/`: a symbolic link which points to the directory which is the most recent release (which may be useful for implementations providing specific entry points for validating against the latest version of the specification)
36
37Inside each version directory there are a number of `.json` files each containing a collection of related tests.
38Often the grouping is by property under test, but not always.
39In addition to the `.json` files, each version directory contains one or more special subdirectories whose purpose is [described below](#subdirectories-within-each-draft), and which contain additional `.json` files.
40
41Each `.json` file consists of a single JSON array of test cases.
42
43### Terminology
44
45For clarity, we first define this document's usage of some testing terminology:
46
47| term            | definition                                                                                                                                                        |
48|-----------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------|
49| **test suite**  | the entirety of the contents of this repository, containing tests for multiple different releases of the JSON Schema specification                                |
50| **test case**   | a single schema, along with a description and an array of *test*s                                                                                                 |
51| **test**        | within a *test case*, a single test example, containing a description, instance and a boolean indicating whether the instance is valid under the test case schema |
52| **test runner** | a program, external to this repository and authored by a user of this suite, which is executing each of the tests in the suite                                    |
53
54An example illustrating this structure is immediately below, and a JSON Schema containing a formal definition of the contents of test cases can be found [alongside this README](./test-schema.json).
55
56### Sample Test Case
57
58Here is a single *test case*, containing one or more tests:
59
60```json
61{
62    "description": "The test case description",
63    "schema": { "type": "string" },
64    "tests": [
65        {
66            "description": "a test with a valid instance",
67            "data": "a string",
68            "valid": true
69        },
70        {
71            "description": "a test with an invalid instance",
72            "data": 15,
73            "valid": false
74        }
75    ]
76}
77```
78
79### Subdirectories Within Each Draft
80
81There is currently only one additional subdirectory that may exist within each draft test directory.
82
83This is:
84
851. `optional/`: Contains tests that are considered optional.
86
87Note, the `optional/` subdirectory today conflates many reasons why a test may be optional -- it may be because tests within a particular file are indeed not required by the specification but still potentially useful to an implementer, or it may be because tests within it only apply to programming languages with particular functionality (in
88which case they are not truly optional in such a language).
89In the future this directory structure will be made richer to reflect these differences more clearly.
90
91## Using the Suite to Test a Validator Implementation
92
93The test suite structure was described [above](#introduction-to-the-test-suite-structure).
94
95If you are authoring a new validator implementation, or adding support for an additional version of the specification, this section describes:
96
971. How to implement a test runner which passes tests to your validator
982. Assumptions the suite makes about how the test runner will configure your validator
993. Invariants the test suite claims to hold for its tests
100
101### How to Implement a Test Runner
102
103Presented here is a possible implementation of a test runner.
104The precise steps described do not need to be followed exactly, but the results of your own procedure should produce the same effects.
105
106To test a specific version:
107
108* For 2019-09 and later published drafts, implementations that are able to detect the draft of each schema via `$schema` SHOULD be configured to do so
109* For draft-07 and earlier, draft-next, and implementations unable to detect via `$schema`, implementations MUST be configured to expect the draft matching the test directory name
110* Load any remote references [described below](additional-assumptions) and configure your implementation to retrieve them via their URIs
111* Walk the filesystem tree for that version's subdirectory and for each `.json` file found:
112
113    * if the file is located in the root of the version directory:
114
115        * for each test case present in the file:
116
117            * load the schema from the `"schema"` property
118            * load (or log) the test case description from the `"description"` property for debugging or outputting
119            * for each test in the `"tests"` property:
120
121                * load the instance to be tested from the `"data"` property
122                * load (or log) the individual test description from the `"description"` property for debugging or outputting
123
124                * use the schema loaded above to validate whether the instance is considered valid under your implementation
125
126                * if the result from your implementation matches the value found in the `"valid"` property, your implementation correctly implements the specific example
127                * if the result does not match, or your implementation errors or crashes, your implementation does not correctly implement the specific example
128
129    * otherwise it is located in a special subdirectory as described above.
130      Follow the additional assumptions and restrictions for the containing subdirectory, then run the test case as above.
131
132If your implementation supports multiple versions, run the above procedure for each version supported, configuring your implementation as appropriate to call each version individually.
133
134### Additional Assumptions
135
1361. The suite, notably in its `refRemote.json` file in each draft, expects a number of remote references to be configured.
137   These are JSON documents, identified by URI, which are used by the suite to test the behavior of the `$ref` keyword (and related keywords).
138   Depending on your implementation, you may configure how to "register" these *either*:
139
140    * by directly retrieving them off the filesystem from the `remotes/` directory, in which case you should load each schema with a retrieval URI of `http://localhost:1234` followed by the relative path from the remotes directory -- e.g. a `$ref` to `http://localhost:1234/foo/bar/baz.json` is expected to resolve to the contents of the file at `remotes/foo/bar/baz.json`
141
142    * or alternatively, by executing `bin/jsonschema_suite remotes` using the executable in the `bin/` directory, which will output a JSON object containing all of the remotes combined, e.g.:
143
144    ```
145
146    $  bin/jsonschema_suite remotes
147    ```
148    ```json
149    {
150        "http://localhost:1234/baseUriChange/folderInteger.json": {
151            "type": "integer"
152        },
153        "http://localhost:1234/baseUriChangeFolder/folderInteger.json": {
154            "type": "integer"
155        }
156    }
157    ```
158
1592. Test cases found within [special subdirectories](#subdirectories-within-each-draft) may require additional configuration to run.
160   In particular, tests within the `optional/format` subdirectory may require implementations to change the way they treat the `"format"`keyword (particularly on older drafts which did not have a notion of vocabularies).
161
162### Invariants & Guarantees
163
164The test suite guarantees a number of things about tests it defines.
165Any deviation from the below is generally considered a bug.
166If you suspect one, please [file an issue](https://github.com/json-schema-org/JSON-Schema-Test-Suite/issues/new):
167
1681. All files containing test cases are valid JSON.
1692. The contents of the `"schema"` property in a test case are always valid
170   JSON Schemas under the corresponding specification.
171
172   The rationale behind this is that we are testing instances in a test's `"data"` element, and not the schema itself.
173   A number of tests *do* test the validity of a schema itself, but do so by representing the schema as an instance inside a test, with the associated meta-schema in the `"schema"` property (via the `"$ref"` keyword):
174
175   ```json
176   {
177       "description": "Test the \"type\" schema keyword",
178       "schema": {
179           "$ref": "https://json-schema.org/draft/2019-09/schema"
180        },
181       "tests": [
182           {
183               "description": "Valid: string",
184               "data": {
185                   "type": "string"
186               },
187               "valid": true
188           },
189           {
190               "description": "Invalid: null",
191               "data": {
192                   "type": null
193               },
194               "valid": false
195           }
196       ]
197   }
198   ```
199   See below for some [known limitations](#known-limitations).
200
201## Known Limitations
202
203This suite expresses its assertions about the behavior of an implementation *within* JSON Schema itself.
204Each test is the application of a schema to a particular instance.
205This means that the suite of tests can test against any behavior a schema can describe, and conversely cannot test against any behavior which a schema is incapable of representing, even if the behavior is mandated by the specification.
206
207For example, a schema can require that a string is a _URI-reference_ and even that it matches a certain pattern, but even though the specification contains [recommendations about URIs being normalized](https://json-schema.org/draft/2020-12/json-schema-core.html#name-the-id-keyword), a JSON schema cannot today represent this assertion within the core vocabularies of the specifications, so no test covers this behavior.
208
209## Who Uses the Test Suite
210
211This suite is being used by:
212
213### Clojure
214
215* [jinx](https://github.com/juxt/jinx)
216* [json-schema](https://github.com/tatut/json-schema)
217
218### Coffeescript
219
220* [jsck](https://github.com/pandastrike/jsck)
221
222### Common Lisp
223
224* [json-schema](https://github.com/fisxoj/json-schema)
225
226### C++
227
228* [Modern C++ JSON schema validator](https://github.com/pboettch/json-schema-validator)
229* [Valijson](https://github.com/tristanpenman/valijson)
230
231### Dart
232
233* [json\_schema](https://github.com/patefacio/json_schema)
234
235### Elixir
236
237* [ex\_json\_schema](https://github.com/jonasschmidt/ex_json_schema)
238
239### Erlang
240
241* [jesse](https://github.com/for-GET/jesse)
242
243### Go
244
245* [gojsonschema](https://github.com/sigu-399/gojsonschema)
246* [validate-json](https://github.com/cesanta/validate-json)
247
248### Haskell
249
250* [aeson-schema](https://github.com/timjb/aeson-schema)
251* [hjsonschema](https://github.com/seagreen/hjsonschema)
252
253### Java
254
255* [json-schema-validator](https://github.com/daveclayton/json-schema-validator)
256* [everit-org/json-schema](https://github.com/everit-org/json-schema)
257* [networknt/json-schema-validator](https://github.com/networknt/json-schema-validator)
258* [Justify](https://github.com/leadpony/justify)
259* [Snow](https://github.com/ssilverman/snowy-json)
260* [jsonschemafriend](https://github.com/jimblackler/jsonschemafriend)
261
262### JavaScript
263
264* [json-schema-benchmark](https://github.com/Muscula/json-schema-benchmark)
265* [direct-schema](https://github.com/IreneKnapp/direct-schema)
266* [is-my-json-valid](https://github.com/mafintosh/is-my-json-valid)
267* [jassi](https://github.com/iclanzan/jassi)
268* [JaySchema](https://github.com/natesilva/jayschema)
269* [json-schema-valid](https://github.com/ericgj/json-schema-valid)
270* [Jsonary](https://github.com/jsonary-js/jsonary)
271* [jsonschema](https://github.com/tdegrunt/jsonschema)
272* [request-validator](https://github.com/bugventure/request-validator)
273* [skeemas](https://github.com/Prestaul/skeemas)
274* [tv4](https://github.com/geraintluff/tv4)
275* [z-schema](https://github.com/zaggino/z-schema)
276* [jsen](https://github.com/bugventure/jsen)
277* [ajv](https://github.com/epoberezkin/ajv)
278* [djv](https://github.com/korzio/djv)
279
280### Node.js
281
282For node.js developers, the suite is also available as an [npm](https://www.npmjs.com/package/@json-schema-org/tests) package.
283
284Node-specific support is maintained in a [separate repository](https://github.com/json-schema-org/json-schema-test-suite-npm) which also welcomes your contributions!
285
286### .NET
287
288* [JsonSchema.Net](https://github.com/gregsdennis/json-everything)
289* [Newtonsoft.Json.Schema](https://github.com/JamesNK/Newtonsoft.Json.Schema)
290
291### Perl
292
293* [Test::JSON::Schema::Acceptance](https://github.com/karenetheridge/Test-JSON-Schema-Acceptance) (a wrapper of this test suite)
294* [JSON::Schema::Modern](https://github.com/karenetheridge/JSON-Schema-Modern)
295* [JSON::Schema::Tiny](https://github.com/karenetheridge/JSON-Schema-Tiny)
296
297### PHP
298
299* [opis/json-schema](https://github.com/opis/json-schema)
300* [json-schema](https://github.com/justinrainbow/json-schema)
301* [json-guard](https://github.com/thephpleague/json-guard)
302
303### PostgreSQL
304
305* [postgres-json-schema](https://github.com/gavinwahl/postgres-json-schema)
306* [is\_jsonb\_valid](https://github.com/furstenheim/is_jsonb_valid)
307
308### Python
309
310* [jsonschema](https://github.com/Julian/jsonschema)
311* [fastjsonschema](https://github.com/seznam/python-fastjsonschema)
312* [hypothesis-jsonschema](https://github.com/Zac-HD/hypothesis-jsonschema)
313* [jschon](https://github.com/marksparkza/jschon)
314* [python-experimental, OpenAPI Generator](https://github.com/OpenAPITools/openapi-generator/blob/master/docs/generators/python-experimental.md)
315
316### Ruby
317
318* [json-schema](https://github.com/hoxworth/json-schema)
319* [json\_schemer](https://github.com/davishmcclurg/json_schemer)
320
321### Rust
322
323* [jsonschema](https://github.com/Stranger6667/jsonschema-rs)
324* [valico](https://github.com/rustless/valico)
325
326### Scala
327
328* [typed-json](https://github.com/frawa/typed-json)
329
330### Swift
331
332* [JSONSchema](https://github.com/kylef/JSONSchema.swift)
333
334If you use it as well, please fork and send a pull request adding yourself to
335the list :).
336
337## Contributing
338
339If you see something missing or incorrect, a pull request is most welcome!
340
341There are some sanity checks in place for testing the test suite. You can run
342them with `bin/jsonschema_suite check` or `tox`. They will be run automatically
343by [GitHub Actions](https://github.com/json-schema-org/JSON-Schema-Test-Suite/actions?query=workflow%3A%22Test+Suite+Sanity+Checking%22)
344as well.
345
346This repository is maintained by the JSON Schema organization, and will be governed by the JSON Schema steering committee (once it exists).
347