You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
c7257e92 refOfUnknownKeyword.json rm unused $id (#890)
bf54c9bd support negative multiples (#888)
e819f329 Fix broken test in v1/ref.json from PR #873
29965c01 Merge pull request #873 from jdesrosiers/fix-v1-meta-schema-ref
3b08f79b test: add some missing edge cases to json-pointer format tests (#877)
a1de34ae test(format): allow consecutive hyphens in hostname (RFC1123) (#874)
aac8a1bd chore: remove duplicate idn-email test and fix ambiguous test descriptions (#881)
78048ead format: add RFC 3339 unknown local offset (-00:00) test for time (#878)
ee3fc3d8 test(draft2020-12): add boundary coverage for maxContains = 0 (#857)
54ed4d1f Add additional URI edge cases and include them in v1 (#850)
87f09d65 Add missing email format edge cases across drafts (#849)
f2720d32 more tests for the duration format: missing units (#875)
e391238a Draft-06/7 don't allow empty enums (#876)
c7fc5095 Add tests for empty enum validation (#870)
20f4897c docs: clarify intent of format-assertion vocabulary test cases (#854)
eb018552 Add annotation tests for $dynamicRef and $dynamicAnchor (#862)
0a1dc5e9 fix: remove duplicate allOf and maxItems entries in draft2020-12 KNOWN set (#865)
06e3102d docs: fix typos and wording in suite documentation (#867)
031e9e9a test(duration): expand RFC 3339 grammar coverage for duration format (#868)
3181cefa Fix issues with v1 tests
06481b14 test: add unicode pattern and patternProperties tests for draft2020-12 (#837)
601aa708 Add exhaustive ABNF-driven tests with RFC 2673 compliance for ipv4 format (#840)
583d7c62 Merge pull request #851 from Anshikakalpana/fix/maximum-test-description-mismatch
61bfe27a fix: apply description fix to v1 maximum test
516d71f7 fix: align test description with actual valid field in maximum.json across drafts
4253477c Merge pull request #836 from VIDIT45AGARWAL/date-time
9e787e38 Add tests for out-of-range time components in date-time format
bce6a47c Merge pull request #793 from 414owen/os/fix-unevaluatedproperties-test-case
a17eb1f6 Merge pull request #833 from JulesGosnell/add-m3-validator
648811b2 Add M3 JSON Schema validator to Who Uses section
32fa6e28 Merge pull request #827 from Shristibot/required-null
086bf9a1 Revert unintended .gitignore change
909b58d7 Remove unrelated files from PR
c957e914 Add boolean and null cases to draft-04 , draft-06 and v1
0650027c Add boolean case to required non-object validation
d8d15620 Reset README to upstream to avoid unrelated changes
f6523449 Add required + null test case for issue #821
75995a1c Merge pull request #818 from fperrad/lua-schema
7e6b3a15 Add lua-schema in README.md
8b826d6b Merge pull request #816 from kpavlov/kotlinx-schema-link
b3ae21c2 Update README.md
646f11da Merge pull request #812 from Shristibot/clarification-on-test-suite
ba3f9505 Update README.md
3fc54179 Merge pull request #808 from json-schema-org/ether/more-fix-remotes
3550ca73 clarify optional tests and document integer example
fac4f6f6 Clarify that the test-suite is not astyle guide
ece5bd10 Merge pull request #804 from Shristibot/add-if-then-else-tests
525adc90 move aside remotes that are not valid in earlier drafts
bbe22f88 remove redundant remotes that also exist in draft-specific directories
ea0baf8d these files are no longer used, and contain an unresolveable $ref on earlier drafts
bef1e2ed Remove redundant then/else false test cases
910af732 Merge pull request #803 from json-schema-org/gregsdennis/proposals
23ff2750 Fix incorrect test condition, simplify test, and copy tests to all drafts as requested
ccfb91af Fix description style and add false-handling tests for issue #767
bc1763ed Fix description formatting in false-handling tests
bd13fcf7 Add if/then/else test cases for issue #767
dc659514 added documentation about folder in root readme and folder readme
c4a99e00 move propertyDependencies tests to proposals folder
d69537ac Merge pull request #802 from json-schema-org/ether/draft4-boolean-schema
79ee846d Merge pull request #801 from json-schema-org/ether/missing-not-tests
2da8f9a6 remove unused remotes files
15cc161f add missing not tests from #714
47ff61c2 Merge pull request #799 from frawa/frawa-patch-1
4a640206 Merge pull request #800 from frawa/frawa-patch-2
fdabd3f9 Merge pull request #795 from anmolbhadoriya5849/fix/issue-768-additionalItems-case
cd02dece Add Unison section to README
4da52d58 Remove 'typed-json' link from README
232f0795 Revert removal of additionalProperties applicator tests in v1
b4032b96 Revert removal of additionalProperties applicator tests in draft2020-12
df833be4 Remove test case for additionalProperties behavior
0d54c828 Remove test for additionalProperties behavior
756eebd6 Remove valid case for additionalItems validation
a3feff79 Remove valid case for additionalItems validation
8ba96359 Remove test for additionalItems applicators
f3488882 Remove valid test case for additionalItems
da4a5684 Remove valid case for additionalItems validation
a247442b Merge pull request #794 from json-schema-org/ether/dynamic-scopes-subschemas
19810ec1 Add a test for respecting dynamic scopes while avoiding the root of each schema
e0e431f8 Fix unevaluatedProperties & additionalProperties test case
8da0a7bc Merge pull request #792 from jdesrosiers/more-v1-transistion
94c0b0e1 Fix more "draft/next" occurances that we missed
8fd4fcec Merge pull request #786 from jdesrosiers/a-label-tests
be58fa98 Merge pull request #791 from json-schema-org/ether/fix-draft-next-removal
acaece38 stop using the term "draft" to mean "version", and fix the remaining mentions of draft-next (now v1)
980e1052 Merge pull request #790 from dylankerr-bis/add-test-sibling-nested-ref-id
b5037ed7 Add test for sibling $ref and $id in nested schema to v1 suite
b8bb8585 Add more comprehensive tessts for A-labels in the "hostname" format
9b8ef536 Cleanup hostname tests (remove dups, fix descriptions, reorder)
4cf55996 Merge pull request #776 from davishmcclurg/idn-hostname-separators
a930db43 Shorten test descriptions to appease ci
aab08752 Updates form PR feedback
081a16a7 Test IDN label separators separate labels
658c8cf9 Test IDN label separator in `hostname` format
38c04b83 Test label separator position in hostname formats
cabbd65c Merge pull request #735 from zaplapl/main
ed74ee74 Add unicode tests to all the dialects
fd0df6cf Update unicode tests per PR feedback
60098220 base test implementation for const equality for strings based on evalutaion by individual codepoint
ef78a0eb Merge pull request #683 from mwadams/main
db365735 Added tests to encoded refs to unknown keywords.
59944b7e Add test for sibling $ref and $id in nested schema to 2019 suite
2196a531 Add test for sibling $ref and $id in nested schema to 2020 suite
e99b24c9 Merge pull request #788 from jdesrosiers/next-to-v1
7eac79be Bring tests up-to-date with format changes
ae27be13 Bring dynamic reference tests up to date with spec changes
79dc2fd3 Unknown optional vocabs can't be ignored anymore
51f84640 Change draft-next to v1
f4e41b0c Merge pull request #785 from jdesrosiers/uneval-nested-conflicts
60a2633f Merge pull request #778 from koplas/extend-uri-validation-tests
bf65ef20 Copy uri tests to other drafts
02a0d89f Merge pull request #743 from Vinit-Pandit/MeastroZI-patch-1
c1545e6c Update unevaluatedProperties.json
7e970016 Merge pull request #764 from Dragonsangel/date_time_validation
8970b1ca Add tests for evaluated property/item nesting conflicts
677a9399 Merge pull request #784 from sangamon/781-date-time-format-extended-year
9672a5b6 Merge pull request #783 from sangamon/780-uuid-format-shifted-dashes
28e49b14 add test for invalid "shifted" dashes in uuid format (#780)
aecf038c add test for invalid extended year in date-time format (#781)
15e4505b Merge pull request #765 from JDepooter/unevaluated_items_with_mincontains_0
b8611c40 Extend URI and URI-reference tests
48461fc3 Merge pull request #775 from jviotti/annotations-content-id
9f7dce32 Merge pull request #766 from json-schema-org/ether/bare-email
a1e077d4 ensure we do not parse email strings permissively
72c670b0 Delete identifiers in Content annotation tests
675186ba Take into account `$id` for annotation schema location assertions
100e9823 Add tests for unevaluatedItems with minContains = 0
0be5549b Added tests to ensure date and time formats don't accept date-time formats
git-subtree-dir: json
git-subtree-split: c7257e92580678a086f0b9243a1903ed88bd27f7
This repository contains a set of JSON objects that implementers of JSON Schema validation libraries can use to test their validators.
11
+
The test suite repository exists to verify specified behavior defined by the JSON Schema specification and should not be confused with a style guide. It is not intended to demonstrate how schemas ought to be written. Tests may appear unusual or unintuitive, but they exist solely to exercise behavior prescribed by the specification.
11
12
12
13
It is meant to be language agnostic and should require only a JSON parser.
13
14
The conversion of the JSON objects into tests within a specific language and test framework of choice is left to be done by the validator implementer.
@@ -16,8 +17,8 @@ The recommended workflow of this test suite is to clone the `main` branch of thi
16
17
17
18
## Coverage
18
19
19
-
All JSON Schema specification releases should be well covered by this suite, including drafts 2020-12, 2019-09, 07, 06, 04 and 03.
20
-
Drafts 04 and 03 are considered "frozen" in that less effort is put in to backport new tests to these versions.
20
+
All JSON Schema specification releases should be well covered by this suite, including versions draft-2020-12, draft-2019-09, draft-07, draft-06, draft-04 and draft-03.
21
+
Versions draft-04 and draft-03 are considered "frozen" in that less effort is put in to backport new tests to these versions.
21
22
22
23
Additional coverage is always welcome, particularly for bugs encountered in real-world implementations.
23
24
If you see anything missing or incorrect, please feel free to [file an issue](https://github.com/json-schema-org/JSON-Schema-Test-Suite/issues) or [submit a PR](https://github.com/json-schema-org/JSON-Schema-Test-Suite).
@@ -31,14 +32,13 @@ Inside that directory is a subdirectory for each released version of the specifi
31
32
32
33
The structure and contents of each file in these directories is described below.
33
34
34
-
In addition to the version-specific subdirectories, two additional directories are present:
35
+
In addition to the version-specific subdirectories, one additional directory is present:
35
36
36
-
1.`draft-next/`: containing tests for the next version of the specification whilst it is in development
37
-
2.`latest/`: a symbolic link which points to the directory which is the most recent release (which may be useful for implementations providing specific entry points for validating against the latest version of the specification)
37
+
1.`latest/`: a symbolic link which points to the directory which is the most recent release (which may be useful for implementations providing specific entry points for validating against the latest version of the specification)
38
38
39
39
Inside each version directory there are a number of `.json` files each containing a collection of related tests.
40
40
Often the grouping is by property under test, but not always.
41
-
In addition to the `.json` files, each version directory contains one or more special subdirectories whose purpose is [described below](#subdirectories-within-each-draft), and which contain additional `.json` files.
41
+
In addition to the `.json` files, each version directory contains one or more special subdirectories whose purpose is [described below](#subdirectories-within-each-version-directory), and which contain additional `.json` files.
42
42
43
43
Each `.json` file consists of a single JSON array of test cases.
44
44
@@ -78,17 +78,15 @@ Here is a single *test case*, containing one or more tests:
78
78
}
79
79
```
80
80
81
-
### Subdirectories Within Each Draft
81
+
### Subdirectories Within Each Version directory
82
82
83
-
There is currently only one additional subdirectory that may exist within each draft test directory.
83
+
A specification version test directory may contain one or more subdirectories.
84
84
85
-
This is:
85
+
These are:
86
86
87
-
1.`optional/`: Contains tests that are considered optional.
87
+
1.`optional/`: Contains tests that are considered optional. Note that this subdirectory currently conflates many reasons why a test may be optional -- it may be because tests within a particular file are indeed not required by the specification but still potentially useful to an implementer, or it may be because tests within it only apply to programming languages with particular functionality (in which case they are not truly optional in such a language). In the future this directory structure will be made richer to reflect these differences more clearly.
88
88
89
-
Note, the `optional/` subdirectory today conflates many reasons why a test may be optional -- it may be because tests within a particular file are indeed not required by the specification but still potentially useful to an implementer, or it may be because tests within it only apply to programming languages with particular functionality (in
90
-
which case they are not truly optional in such a language).
91
-
In the future this directory structure will be made richer to reflect these differences more clearly.
89
+
2.`proposals/`: Contains a subfolder for each active proposal to the specification. If the proposal is a keyword (generally the case), then the subfolder will bear the name of that keyword. Inside the proposal subfolder is a series of test files that would contain amendments to the required test suite should the proposal be incorporated into the specification. These tests should be considered volatile while the proposal is in development; however, implementations claiming to support the proposal are expected to pass its tests.
92
90
93
91
## Using the Suite to Test a Validator Implementation
94
92
@@ -107,8 +105,8 @@ The precise steps described do not need to be followed exactly, but the results
107
105
108
106
To test a specific version:
109
107
110
-
* For 2019-09 and later published drafts, implementations that are able to detect the draft of each schema via `$schema` SHOULD be configured to do so
111
-
* For draft-07 and earlier, draft-next, and implementations unable to detect via `$schema`, implementations MUST be configured to expect the draft matching the test directory name
108
+
* For 2019-09 and later published versions, implementations that are able to detect the version of each schema via `$schema` SHOULD be configured to do so
109
+
* For draft-07 and earlier, v1 (not yet released), and implementations unable to detect via `$schema`, implementations MUST be configured to expect the version matching the test directory name
112
110
* Load any remote references [described below](#additional-assumptions) and configure your implementation to retrieve them via their URIs
113
111
* Walk the filesystem tree for that version's subdirectory and for each `.json` file found:
114
112
@@ -135,7 +133,7 @@ If your implementation supports multiple versions, run the above procedure for e
135
133
136
134
### Additional Assumptions
137
135
138
-
1. The suite, notably in its `refRemote.json` file in each draft, expects a number of remote references to be configured.
136
+
1. The suite, notably in its `refRemote.json` file in each specification version directory, expects a number of remote references to be configured.
139
137
These are JSON documents, identified by URI, which are used by the suite to test the behavior of the `$ref` keyword (and related keywords).
140
138
Depending on your implementation, you may configure how to "register" these *either*:
141
139
@@ -158,7 +156,7 @@ If your implementation supports multiple versions, run the above procedure for e
158
156
}
159
157
```
160
158
161
-
2. Test cases found within [special subdirectories](#subdirectories-within-each-draft) may require additional configuration to run.
159
+
2. Test cases found within [special subdirectories](#subdirectories-within-each-version-directory) may require additional configuration to run.
162
160
In particular, when running tests within the `optional/format` subdirectory, test runners should configure implementations to enable format validation, where the implementation supports it.
*[M3](https://github.com/JulesGosnell/m3) (all drafts, pure Clojure — also usable from Java, Kotlin, and Scala)
284
285
285
286
### Kotlin
286
287
287
288
*[json-schema-validation-comparison](https://www.creekservice.org/json-schema-validation-comparison/functional) (Comparison site for JVM-based validator implementations)
@@ -335,13 +342,17 @@ Node-specific support is maintained in a [separate repository](https://github.co
335
342
### Scala
336
343
337
344
*[json-schema-validation-comparison](https://www.creekservice.org/json-schema-validation-comparison/functional) (Comparison site for JVM-based validator implementations)
"markdownDescription": "This schema shouldn't include `$schema` or `id`/`$id` unless necesary for the test because Test Cases should be designed to work with as many releases as possible.",
16
+
"markdownDescription": "This schema shouldn't include `$schema` or `id`/`$id` unless necessary for the test because Test Cases should be designed to work with as many releases as possible.",
Copy file name to clipboardExpand all lines: output-tests/README.md
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -8,7 +8,7 @@ _**NOTE** Although the formats didn't change much between 2019-09 and 2020-12, t
8
8
9
9
The tests are organized by specification release and then into two categories: content and structure.
10
10
11
-
Content tests verify that the keywords are producing the correct annotations and/or error messages. Since there are no requirements on the content of error messages, there's not much that can be verified for them, but it is possible to identify when a error message _could_ be present. Primarily, these tests need to extensively cover the annotation behaviors of each keyword. The only output format needed for these tests is `basic` for 2019-09/2020-12 and `list` for later versions.
11
+
Content tests verify that the keywords are producing the correct annotations and/or error messages. Since there are no requirements on the content of error messages, there's not much that can be verified for them, but it is possible to identify when an error message _could_ be present. Primarily, these tests need to extensively cover the annotation behaviors of each keyword. The only output format needed for these tests is `basic` for 2019-09/2020-12 and `list` for later versions.
12
12
13
13
Structure tests verify that the structures of the various formats (i.e. `flag`, `basic`, `detailed`, `verbose` for 2019-09/2020-12 and `flag`, `list`, `hierarchical` for later versions) are correct. These tests don't need to cover each keyword; rather they need to sufficiently cover the various aspects of building the output structures by using whatever keywords are necessary to do so.
14
14
@@ -22,7 +22,7 @@ The `output` property itself has a property for each of the output formats where
22
22
23
23
## Other notes
24
24
25
-
### Ambiguity around 2020-09/2020-12 `basic`
25
+
### Ambiguity around 2019-09/2020-12 `basic`
26
26
27
27
The 2019-09/2020-12 specs don't define the structure of `basic` very thoroughly. Specifically there is a nuance where if the list contains a single output node, there are two possible structures, given the text:
0 commit comments