Skip to content

Commit

Permalink
Resolves improper anchoring of patternProperties (#783)
Browse files Browse the repository at this point in the history
* Resolves improper anchoring of patternProperties

Fixes #782

* Resolves improper anchoring of regular expressions in both Joni and JDK engines.

Resolves #495 and #782

---------

Co-authored-by: Faron Dutton <faron.dutton@insightglobal.com>
  • Loading branch information
fdutton and Faron Dutton authored May 23, 2023
1 parent 77cd232 commit f09740a
Show file tree
Hide file tree
Showing 6 changed files with 274 additions and 75 deletions.
147 changes: 84 additions & 63 deletions doc/compatibility.md
Original file line number Diff line number Diff line change
@@ -1,76 +1,97 @@

### Legend

Symbol | Meaning |
:-----:|---------|
🟢 | Fully implemented
🟡 | Partially implemented
🔴 | Not implemented
🚫 | Not defined in Schema Version.
| Symbol | Meaning |
|:------:|:----------------------|
| 🟢 | Fully implemented |
| 🟡 | Partially implemented |
| 🔴 | Not implemented |
| 🚫 | Not defined |

### Compatibility with JSON Schema versions

Validation Keyword/Schema | Draft 4 | Draft 6 | Draft 7 | Draft 2019-09 |
---------------- |:--------------:|:-------: |:-------: |:-------------:|
$ref | 🟢 | 🟢 | 🟢 | 🟢
additionalProperties | 🟢 | 🟢 | 🟢 | 🟢
additionalItems | 🟢 | 🟢 | 🟢 | 🟢
allOf | 🟢 | 🟢 | 🟢 | 🟢
anyOf | 🟢 | 🟢 | 🟢 | 🟢
const | 🚫 | 🟢 | 🟢 | 🟢
contains | 🚫 | 🟢 | 🟢 | 🟢
contentEncoding | 🚫 | 🚫 | 🔴 | 🔴
contentMediaType | 🚫 | 🚫 | 🔴 | 🔴
dependencies | 🟢 | 🟢 |🟢 | 🟢
enum | 🟢 | 🟢 | 🟢 | 🟢
exclusiveMaximum (boolean) | 🟢 | 🚫 | 🚫 | 🚫
exclusiveMaximum (numeric) | 🚫 | 🟢 | 🟢 | 🟢
exclusiveMinimum (boolean) | 🟢 | 🚫 | 🚫 | 🚫
exclusiveMinimum (numeric) | 🚫 | 🟢 | 🟢 | 🟢
items | 🟢 | 🟢 | 🟢 | 🟢
maximum | 🟢 | 🟢 | 🟢 | 🟢
maxItems | 🟢 | 🟢 | 🟢 | 🟢
maxLength | 🟢 | 🟢 | 🟢 | 🟢
maxProperties | 🟢 | 🟢 | 🟢 | 🟢
minimum | 🟢 | 🟢 | 🟢 | 🟢
minItems | 🟢 | 🟢 | 🟢 | 🟢
minLength | 🟢 | 🟢 | 🟢 | 🟢
minProperties | 🟢 | 🟢 | 🟢 | 🟢
multipleOf | 🟢 | 🟢 | 🟢 | 🟢
not | 🟢 | 🟢 | 🟢 | 🟢
oneOf | 🟢 | 🟢 | 🟢 | 🟢
pattern | 🟢 | 🟢 | 🟢 | 🟢
patternProperties | 🟢 | 🟢 | 🟢 | 🟢
properties | 🟢 | 🟢 | 🟢 | 🟢
propertyNames | 🚫 | 🔴 | 🔴 | 🔴
required | 🟢 | 🟢 | 🟢 | 🟢
type | 🟢 | 🟢 | 🟢 | 🟢
uniqueItems | 🟢 | 🟢 | 🟢 | 🟢
| Keyword | Draft 4 | Draft 6 | Draft 7 | Draft 2019-09 | Draft 2020-12 |
|:---------------------------|:-------:|:-------:|:-------:|:-------------:|:-------------:|
| $anchor | 🚫 | 🚫 | 🚫 | 🔴 | 🔴 |
| $dynamicAnchor | 🚫 | 🚫 | 🚫 | 🚫 | 🔴 |
| $dynamicRef | 🚫 | 🚫 | 🚫 | 🚫 | 🔴 |
| $id | 🟡 | 🟡 | 🟡 | 🟡 | 🟡 |
| $recursiveAnchor | 🚫 | 🚫 | 🚫 | 🔴 | 🔴 |
| $recursiveRef | 🚫 | 🚫 | 🚫 | 🔴 | 🔴 |
| $ref | 🟡 | 🟡 | 🟡 | 🟡 | 🟡 |
| $vocabulary | 🚫 | 🚫 | 🚫 | 🔴 | 🔴 |
| additionalItems | 🟢 | 🟢 | 🟢 | 🟢 | 🟢 |
| additionalProperties | 🟢 | 🟢 | 🟢 | 🟢 | 🟢 |
| allOf | 🟢 | 🟢 | 🟢 | 🟢 | 🟢 |
| anyOf | 🟢 | 🟢 | 🟢 | 🟢 | 🟢 |
| const | 🚫 | 🟢 | 🟢 | 🟢 | 🟢 |
| contains | 🚫 | 🟢 | 🟢 | 🟢 | 🟢 |
| contentEncoding | 🚫 | 🚫 | 🔴 | 🔴 | 🔴 |
| contentMediaType | 🚫 | 🚫 | 🔴 | 🔴 | 🔴 |
| contentSchema | 🚫 | 🚫 | 🚫 | 🔴 | 🔴 |
| definitions | 🟢 | 🟢 | 🟢 | 🚫 | 🚫 |
| defs | 🚫 | 🚫 | 🚫 | 🟢 | 🟢 |
| dependencies | 🟢 | 🟢 | 🟢 | 🚫 | 🚫 |
| dependentRequired | 🚫 | 🚫 | 🚫 | 🟢 | 🟢 |
| dependentSchemas | 🚫 | 🚫 | 🚫 | 🟢 | 🟢 |
| enum | 🟢 | 🟢 | 🟢 | 🟢 | 🟢 |
| exclusiveMaximum (boolean) | 🟢 | 🚫 | 🚫 | 🚫 | 🚫 |
| exclusiveMaximum (numeric) | 🚫 | 🟢 | 🟢 | 🟢 | 🟢 |
| exclusiveMinimum (boolean) | 🟢 | 🚫 | 🚫 | 🚫 | 🚫 |
| exclusiveMinimum (numeric) | 🚫 | 🟢 | 🟢 | 🟢 | 🟢 |
| if-then-else | 🚫 | 🚫 | 🟢 | 🟢 | 🟢 |
| items | 🟢 | 🟢 | 🟢 | 🟢 | 🟢 |
| maxContains | 🚫 | 🚫 | 🚫 | 🟢 | 🟢 |
| minContains | 🚫 | 🚫 | 🚫 | 🟢 | 🟢 |
| maximum | 🟢 | 🟢 | 🟢 | 🟢 | 🟢 |
| maxItems | 🟢 | 🟢 | 🟢 | 🟢 | 🟢 |
| maxLength | 🟢 | 🟢 | 🟢 | 🟢 | 🟢 |
| maxProperties | 🟢 | 🟢 | 🟢 | 🟢 | 🟢 |
| minimum | 🟢 | 🟢 | 🟢 | 🟢 | 🟢 |
| minItems | 🟢 | 🟢 | 🟢 | 🟢 | 🟢 |
| minLength | 🟢 | 🟢 | 🟢 | 🟢 | 🟢 |
| minProperties | 🟢 | 🟢 | 🟢 | 🟢 | 🟢 |
| multipleOf | 🟢 | 🟢 | 🟢 | 🟢 | 🟢 |
| not | 🟢 | 🟢 | 🟢 | 🟢 | 🟢 |
| oneOf | 🟢 | 🟢 | 🟢 | 🟢 | 🟢 |
| pattern | 🟢 | 🟢 | 🟢 | 🟢 | 🟢 |
| patternProperties | 🟢 | 🟢 | 🟢 | 🟢 | 🟢 |
| prefixItems | 🚫 | 🚫 | 🚫 | 🚫 | 🟢 |
| properties | 🟢 | 🟢 | 🟢 | 🟢 | 🟢 |
| propertyNames | 🚫 | 🟢 | 🟢 | 🟢 | 🟢 |
| readOnly | 🚫 | 🚫 | 🔴 | 🔴 | 🔴 |
| required | 🟢 | 🟢 | 🟢 | 🟢 | 🟢 |
| type | 🟢 | 🟢 | 🟢 | 🟢 | 🟢 |
| unevaluatedItems | 🚫 | 🚫 | 🚫 | 🟢 | 🟢 |
| unevaluatedProperties | 🚫 | 🚫 | 🚫 | 🟢 | 🟢 |
| uniqueItems | 🟢 | 🟢 | 🟢 | 🟢 | 🟢 |
| writeOnly | 🚫 | 🚫 | 🔴 | 🔴 | 🔴 |

### Semantic Validation (Format)

Format | Draft 4 | Draft 6 | Draft 7 | Draft 2019-09 |
-------|---------|---------|---------|---------------|
date |🚫 | 🚫 | 🟢 | 🟢
date-time | 🟢 | 🟢 | 🟢 | 🟢
duration | 🚫 | 🚫 | 🔴 | 🔴
email | 🟢 | 🟢 | 🟢 | 🟢
hostname | 🟢 | 🟢 | 🟢 | 🟢
idn-email | 🚫 | 🚫 | 🔴 | 🔴
idn-hostname | 🚫 | 🚫 | 🔴 | 🔴
ipv4 | 🟢 | 🟢 | 🟢 | 🟢
ipv6 | 🟢 | 🟢 | 🟢 | 🟢
iri | 🚫 | 🚫 | 🔴 | 🔴
iri-reference | 🚫 | 🚫 | 🔴 | 🔴
json-pointer | 🚫 | 🔴 | 🔴 | 🔴
relative-json-pointer | 🚫 | 🔴 | 🔴 | 🔴
regex | 🚫 | 🚫 | 🔴 | 🔴
time | 🚫 | 🚫 | 🟢 | 🟢
uri | 🟢 | 🟢 | 🟢 | 🟢
uri-reference | 🚫 | 🔴 | 🔴 | 🔴
uri-template | 🚫 | 🔴 | 🔴 | 🔴
uuid | 🚫 | 🚫 | 🟢 | 🟢
| Format | Draft 4 | Draft 6 | Draft 7 | Draft 2019-09 | Draft 2020-12 |
|:----------------------|:-------:|:-------:|:-------:|:-------------:|:-------------:|
| date | 🚫 | 🚫 | 🟢 | 🟢 | 🟢 |
| date-time | 🟢 | 🟢 | 🟢 | 🟢 | 🟢 |
| duration | 🚫 | 🚫 | 🚫 | 🟢 | 🟢 |
| email | 🟢 | 🟢 | 🟢 | 🟢 | 🟢 |
| hostname | 🟢 | 🟢 | 🟢 | 🟢 | 🟢 |
| idn-email | 🚫 | 🚫 | 🟢 | 🟢 | 🟢 |
| idn-hostname | 🚫 | 🚫 | 🟢 | 🟢 | 🟢 |
| ipv4 | 🟢 | 🟢 | 🟢 | 🟢 | 🟢 |
| ipv6 | 🟢 | 🟢 | 🟢 | 🟢 | 🟢 |
| iri | 🚫 | 🚫 | 🟢 | 🟢 | 🟢 |
| iri-reference | 🚫 | 🚫 | 🟢 | 🟢 | 🟢 |
| json-pointer | 🚫 | 🟢 | 🟢 | 🟢 | 🟢 |
| relative-json-pointer | 🚫 | 🟢 | 🟢 | 🟢 | 🟢 |
| regex | 🚫 | 🚫 | 🟢 | 🟢 | 🟢 |
| time | 🚫 | 🚫 | 🟢 | 🟢 | 🟢 |
| uri | 🟢 | 🟢 | 🟢 | 🟢 | 🟢 |
| uri-reference | 🚫 | 🟢 | 🟢 | 🟢 | 🟢 |
| uri-template | 🚫 | 🟢 | 🟢 | 🟢 | 🟢 |
| uuid | 🚫 | 🚫 | 🟢 | 🟢 | 🟢 |

### Footnotes
1. Note that the validation are only optional for some of the keywords/formats.
2. Refer to the corresponding JSON schema for more information on whether the keyword/format is optional or not.

Original file line number Diff line number Diff line change
@@ -1,17 +1,23 @@
package com.networknt.schema.regex;

import java.util.regex.Matcher;
import java.util.regex.Pattern;

class JDKRegularExpression implements RegularExpression {
private final Pattern pattern;
private final boolean hasStartAnchor;
private final boolean hasEndAnchor;

JDKRegularExpression(String regex) {
this.pattern = Pattern.compile(regex);
this.hasStartAnchor = '^' == regex.charAt(0);
this.hasEndAnchor = '$' == regex.charAt(regex.length() - 1);
}

@Override
public boolean matches(String value) {
return this.pattern.matcher(value).matches();
Matcher matcher = this.pattern.matcher(value);
return matcher.find() && (!this.hasStartAnchor || 0 == matcher.start()) && (!this.hasEndAnchor || matcher.end() == value.length());
}

}
20 changes: 10 additions & 10 deletions src/test/java/com/networknt/schema/AbstractJsonSchemaTestSuite.java
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@
import static org.junit.jupiter.api.DynamicTest.dynamicTest;

public abstract class AbstractJsonSchemaTestSuite extends HTTPServiceSupport {
protected static final TypeReference<List<TestCase>> testCaseType = new TypeReference<List<TestCase>>() {};
protected static final TypeReference<List<TestCase>> testCaseType = new TypeReference<List<TestCase>>() { /* intentionally empty */};
protected static final Map<String, VersionFlag> supportedVersions = new HashMap<>();
static {
supportedVersions.put("draft2019-09", VersionFlag.V201909);
Expand Down Expand Up @@ -99,7 +99,7 @@ private DynamicNode buildContainer(VersionFlag defaultVersion, TestCase testCase
String msg = e.getMessage();
if (msg.endsWith("' is unrecognizable schema")) {
return dynamicContainer(testCase.getDisplayName(), unsupportedMetaSchema(testCase));
};
}
throw e;
}
}
Expand All @@ -109,7 +109,7 @@ private JsonSchemaFactory buildValidatorFactory(VersionFlag defaultVersion, Test
JsonSchemaFactory base = JsonSchemaFactory.getInstance(specVersion);
return JsonSchemaFactory
.builder(base)
.objectMapper(mapper)
.objectMapper(this.mapper)
.addUriTranslator(URITranslator.combine(
URITranslator.prefix("https://", "http://"),
URITranslator.prefix("http://json-schema.org", "resource:")
Expand All @@ -124,21 +124,21 @@ private DynamicNode buildTest(JsonSchemaFactory validatorFactory, TestSpec testS

SchemaValidatorsConfig config = new SchemaValidatorsConfig();
config.setTypeLoose(typeLoose);
config.setEcma262Validator(true);
config.setEcma262Validator(TestSpec.RegexKind.JDK != testSpec.getRegex());
testSpec.getStrictness().forEach(config::setStrict);
URI testCaseFileUri = URI.create("classpath:" + toForwardSlashPath(testSpec.getTestCase().getSpecification()));
JsonSchema schema = validatorFactory.getSchema(testCaseFileUri, testSpec.getTestCase().getSchema(), config);

return dynamicTest(testSpec.getDescription(), () -> executeAndReset(schema, testSpec));
}

private String toForwardSlashPath(Path file) {
private static String toForwardSlashPath(Path file) {
return file.toString().replace('\\', '/');
}

// For 2019-09 and later published drafts, implementations that are able to
// detect the draft of each schema via $schema SHOULD be configured to do so
private VersionFlag detectVersion(TestCase testCase, VersionFlag defaultVersion) {
private static VersionFlag detectVersion(TestCase testCase, VersionFlag defaultVersion) {
return Stream.of(
detectOptionalVersion(testCase.getSchema()),
detectVersionFromPath(testCase.getSpecification())
Expand All @@ -152,7 +152,7 @@ private VersionFlag detectVersion(TestCase testCase, VersionFlag defaultVersion)
// For draft-07 and earlier, draft-next, and implementations unable to
// detect via $schema, implementations MUST be configured to expect the
// draft matching the test directory name
private Optional<VersionFlag> detectVersionFromPath(Path path) {
private static Optional<VersionFlag> detectVersionFromPath(Path path) {
return StreamSupport.stream(path.spliterator(), false)
.map(Path::toString)
.map(supportedVersions::get)
Expand All @@ -168,7 +168,7 @@ private void executeAndReset(JsonSchema schema, TestSpec testSpec) {
}
}

private void executeTest(JsonSchema schema, TestSpec testSpec) {
private static void executeTest(JsonSchema schema, TestSpec testSpec) {
Set<ValidationMessage> errors = schema.validate(testSpec.getData());

if (testSpec.isValid()) {
Expand Down Expand Up @@ -246,7 +246,7 @@ private List<Path> findTestCases(String basePath) {

private Stream<TestCase> loadTestCases(Path testCaseFile) {
try (InputStream in = new FileInputStream(testCaseFile.toFile())) {
return mapper.readValue(in, testCaseType)
return this.mapper.readValue(in, testCaseType)
.stream()
.peek(testCase -> testCase.setSpecification(testCaseFile))
.filter(this::enabled);
Expand All @@ -258,7 +258,7 @@ private Stream<TestCase> loadTestCases(Path testCaseFile) {
}
}

private Iterable<? extends DynamicNode> unsupportedMetaSchema(TestCase testCase) {
private static Iterable<? extends DynamicNode> unsupportedMetaSchema(TestCase testCase) {
return Collections.singleton(
dynamicTest("Detected an unsupported schema", () -> {
String schema = testCase.getSchema().asText();
Expand Down
18 changes: 17 additions & 1 deletion src/test/java/com/networknt/schema/TestSpec.java
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,11 @@ public class TestSpec {
*/
private final boolean typeLoose;

/**
* Identifies the regular expression engine to use for this test-case.
*/
private final RegexKind regex;

/**
* The TestCase that contains this TestSpec.
*/
Expand All @@ -107,7 +112,8 @@ public TestSpec(
@JsonProperty("strictness") Map<String, Boolean> strictness,
@JsonProperty("validationMessages") Set<String> validationMessages,
@JsonProperty("isTypeLoose") Boolean isTypeLoose,
@JsonProperty("disabled") Boolean disabled
@JsonProperty("disabled") Boolean disabled,
@JsonProperty(value = "regex", defaultValue = "unspecified") RegexKind regex
) {
this.description = description;
this.comment = comment;
Expand All @@ -116,6 +122,7 @@ public TestSpec(
this.validationMessages = validationMessages;
this.disabled = Boolean.TRUE.equals(disabled);
this.typeLoose = Boolean.TRUE.equals(isTypeLoose);
this.regex = regex;
if (null != strictness) {
this.strictness.putAll(strictness);
}
Expand Down Expand Up @@ -211,4 +218,13 @@ public boolean isTypeLoose() {
return typeLoose;
}

public RegexKind getRegex() {
return this.regex;
}

public static enum RegexKind {
@JsonProperty("unspecified") UNSPECIFIED,
@JsonProperty("ecma-262") JONI,
@JsonProperty("jdk") JDK
}
}
64 changes: 64 additions & 0 deletions src/test/resources/draft2020-12/issue495.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
[
{
"description": "issue495 using ECMA-262",
"regex": "ecma-262",
"schema": {
"$schema": "https://json-schema.org/draft/2020-12/schema",
"pattern": "^[a-z]{1,10}$",
"unevaluatedProperties": false
},
"tests": [
{
"description": "an expected property name",
"data": { "aaa": 3 },
"valid": true
},
{
"description": "trailing newline",
"data": { "aaa\n": 3 },
"valid": false
},
{
"description": "embedded newline",
"data": { "aaa\nbbb": 3 },
"valid": false
},
{
"description": "leading newline",
"data": { "\nbbb": 3 },
"valid": false
}
]
},
{
"description": "issue495 using Java Pattern",
"regex": "jdk",
"schema": {
"$schema": "https://json-schema.org/draft/2020-12/schema",
"pattern": "^[a-z]{1,10}$",
"unevaluatedProperties": false
},
"tests": [
{
"description": "an expected property name",
"data": { "aaa": 3 },
"valid": true
},
{
"description": "trailing newline",
"data": { "aaa\n": 3 },
"valid": false
},
{
"description": "embedded newline",
"data": { "aaa\nbbb": 3 },
"valid": false
},
{
"description": "leading newline",
"data": { "\nbbb": 3 },
"valid": false
}
]
}
]
Loading

0 comments on commit f09740a

Please sign in to comment.