oneOf incorrectly evaluates with a error result of Value not allowed even when it is valid

[davidburtonmr]

With a schema of:

{
  "type": "string",
  "oneOf": [
    { "const": "foo" },
    { "const": "bar" }
  ]
}

If you pass foo or bar it correctly evaluate()s to true, but there is still a RuntimeEvaluationResult where error is Value is not allowed.
The current unit tests for oneOf only cover whether it evaluates to true or false and not the returned results.

[ro-pi]

Hi David,

thank you for reporting.

The behavior you describe is actually a design decision and not a bug.
In applicator keywords (like oneOf, allOf, anyOf, ...) JSON schema evaluator will evaluate against all defined schemas of applicator keyword. For example, if you pass "bar", the JSON schema evaluator first evaluates the schema const "foo", which is invalid, and then evaluates the schema "bar", which is valid, which in turn makes the entire oneOf keyword valid.

The RuntimeEvaluationResult is currently as verbose as possible in order to be able to understand for debugging purposes exactly why the oneOf schema is valid (in other words, to see exactly which scheme was considered valid).

If you want to see whether an error is a "real" error in the result, you can also check the "suppressAnnotation" property, which is true if the error and/or annotations were discarded during the evaluation.

It could be useful to implement an option in the output classes that can control the level of result verbosity.

Best regards,
Robert

[ro-pi]

Here is a code example to filter real errors (I have updated the README too):

foreach ($results as $result) {
    /** @var $result \Ropi\JsonSchemaEvaluator\EvaluationContext\RuntimeEvaluationResult */
    if ($result->error && !$result->suppressAnnotation) {
        echo "Error keyword location: '{$result->keywordLocation}'\n";
        echo "Error instance location: '{$result->instanceLocation}'\n";
        echo "Error message: {$result->error}\n";
    }
}

[davidburtonmr]

@ro-pi Thank you for the quick response and detailed explanation. I have incorporated checking suppressAnnotation into my implementation.

[davidburtonmr]

@ro-pi Apologies, but suppressAnnotation is true in all cases. With:

foreach ($results as $result) {
    if ($result->error) {
        echo "Error keyword location: '{$result->keywordLocation}'\n";
        echo "Error instance location: '{$result->instanceLocation}'\n";
        echo "Error message: {$result->error}\n";
        echo "Annotation suppressed: {$result->suppressAnnotation}";
        echo "\n\n";
    }
}

If I pass baz to the example above, I get:

Error keyword location: '/oneOf'
Error instance location: ''
Error message: Value must match exactly one schema, but matches 0
Annotation suppressed: 1

Error keyword location: '/oneOf/const'
Error instance location: ''
Error message: Value not allowed.
Annotation suppressed: 1

Error keyword location: '/oneOf/1/const'
Error instance location: ''
Error message: Value not allowed.
Annotation suppressed: 1

If I pass bar or foo I get:

Error keyword location: '/oneOf/const'
Error instance location: ''
Error message: Value not allowed.
Annotation suppressed: 1

[ro-pi]

@davidburtonmr I see, you are right. In this case, of course, all annotations are suppressed. So we need a new property to recognize "real" errors that actually have an influence on the final result.
I'll have a look in the next few days to see if I can find a good solution.

[ro-pi]

@davidburtonmr I've solved this issue and just created a new release (v0.4.0), which now implements a "type" property in RuntimeEvaluationResult, which can be of value "error" or "annotation". The results with the type "error" are obviously actual errors and results with the type "annotation" are informative for debugging purposes in order to be able to understand which schemas and keywords were validated (and how).

Best regards,
Robert

[davidburtonmr]

@ro-pi Thanks for the fix and a new release, I can confirm that valid values for anyOf do not have any Results where type is error.

One notable is I am not getting Results with type error from e.g. requireds or type validations. My guess that only the lowest level error result (usually an allOf in my case) is given type error - which may be exactly what you intended.

The schemas I am working with are pretty complicated and not really suitable as an example here, if it would help I can provide a simpler example with result outcomes.

[ro-pi]

@davidburtonmr If I have understood you correctly, then the behavior you have described is intentional. The required keyword itself returns error results itself, but not if they are nested insided an applicator keyword.

The reason is, because the sub-schema-validation results are not affecting the overall validation result (e.g. if you have the oneOf-Keyword with 3 sub-schemas and the first 2 sub-schemas are failing, but the last one matches, then validation will complete and the first 2 failed validations are not errors, but annotations).

The allOf keyword is a special case here, because logically all schemas must be valid. I am currently unsure whether I should actually generate error results instead of annotation results in this case to simplify error handling, or whether I should leave it as it is for consistency with the other applicator keywords 🤔