Ecosystem fragmentation

[awwright]

Over at ietf-wg-httpapi/mediatypes#32 (comment), @jdesrosiers wrote:

it's unfortunate that JSON Schema has become fragmented

I have a more positive outlook, could you please elaborate on how you figure this?

Based on the test suite, I would say the JSON Schema ecosystem is as unified as it's ever been. The reason that I and other authors picked up work on a 4-year-old draft was to iron out incompatibilities, differences between implementations, and create a forum for implementors. And we've largely accomplished that. As far as the test suite measures, we have more implementations at 100% compliance with the latest publications than we ever had for draft-04.

So if after all this, there's really still fragmentation, let's take a new look at that. Take for example "uniqueItems" — since this an O(n²) operation, maybe some implementations with limited memory legitimately need to "fragment" from the standard behavior.

If different implementations legitimately need to vary in behavior like this, let's make sure (1) there's a mechanism where they can declare they are varying from that behavior; and (2) work to make sure that significant dimensions of fragmentation are rolled back into the spec.

① For the sake of this first point, assume there should be a single specification that all implementations try their best to follow. Now from time to time, someone will use JSON Schema for something that we couldn't possibly have imagined. For example, a validator with a fixed amount of working memory, a UI generator, code generator, as a sub-language inside another (OpenAPI, or a new language for converting XML to JSON, etc).

Because of the implementation's features or constraints, it may need to extend or violate the spec (e.g. write new keywords, or omit some).

These implementations need a way to declare "I accept a variant that has extensions and/or is incomplete in some fashion". This declaration must be permissionless, so that people can write one-off proof-of-concepts. It needs to work in such that, if the idea takes off, any schemas written for these proof-of-concepts don't surprise other implementations, or vice-versa.

This will require some sort of extension indicator ($schema) and standardized handling of its values (when errors must be generated, and how to handle unknown values, or missing values). Currently, we don't specify error handling or forward compatibility for this case whatsoever.

② calls back to our goal as a forum for implementations to converge: If a variation becomes significantly popular, we may need to write support for these variations into the spec. There's also implementations whose "variation from the spec" is they cannot support extensions or variations at all, these implementations need to be supported too.

For example, suppose several databases cannot support "uniqueItems", so they each declare a variation that excludes this keyword (among other modifications). We should "factor this out" and work it into the spec in such a way it doesn't require any special declaration, but is satisfied with appropriate error handling.

In the case of "uniqueItems", this might look like:

(a) implementations don't have to support every keyword, but they must bail if an unsupported keyword is present. Now, the set of schemas supported by our database are not just those that declare a need for the variant—they are all of the schemas that don't use "uniqueItems". And implementations don't need any special knowledge of the database-specific dialect in order to be compatible with it.

And (b) If the array items are provided in sorting order, "uniqueItems" can be implemented with good memory guarantees. We could introduce a new keyword that requires a specific sorting order. Assume implementations can notice this keyword is an unsupported validation keyword, and so will bail.

This is the same outcome as implementations that don't support the database-specific dialect, so this is not a problem. The difference is now we have a standard keyword that can be used for multiple purposes by most implementations, whereas the database dialect was implementation-specific and unlikely to see wide adoption.

① is the scorched-earth, last-resort option where you can implement your extension right now, but only for your implementation, maybe a small number of implementations. But it's quick and cheap to deploy. ② is the option where we describe compatibility on a finer level, which means more implementations can support it (or decide how to support it in a way right for them), which maximizes cross-platform compatibility (but it needs to go through the standards process).

---

Perhaps first, let's make a list of the dimensions in which the ecosystem is currently fragmented. This list will illustrate where the biggest cross-platform compatibility problems are, and influence what parts of the spec to look at first.

I would like to see instances in the wild, but also, any theoretical examples that might be roadblocked because of compatibility problems.

[handrews]

There have been other proposals to shift the granularity more towards individual keywords rather than the current vocabulary groupings.

This will require some sort of extension indicator ($schema) and standardized handling of its values (when errors must be generated, and how to handle unknown values, or missing values). Currently, we don't specify error handling or forward compatibility for this case whatsoever.

That's not true. The value of $schema is only important in that it tells you how to get to $vocabulary, which has well-defined error behavior. It is only the lack of $schema that produces undefined behavior. As long as the meta-schema is provided (in advance or locatable at runtime):

• if it uses $vocabulary the behavior is well-defined:
  • if all vocabularies are understood, it MUST process the schema
  • if all true vocabularies are understood, but one or more false vocabularies are not, the implementation chooses to either process the schema or not (but it MUST choose from those two options, not undefined behavior)
  • if at least one true vocabulary is not understood, it MUST refuse to process the schema
• if it does not use $vocabulary but the implementation assumes 2020-12 (or 2019-09), then an implementation specializing in validation (which is what we usually mean by "implementation") SHOULD assume the standard vocabularies
• if it does not use $vocabulary but the implementation assumes draft-07 or earlier, then we're using a spec that doesn't have the language about unknown meta-schema behavior being implementation-defined
• if it does not use $vocabulary and the implementation does not assume a particular draft, it's hard to say what spec language controls the behavior, so admittedly I'm not quite sure how to characterize this scenario- it might also be undefined

Anyway, the uniqueItems example is a good one, @awwright , and a compelling argument for a more flexible approach to some aspect of this. But what we have now can be tweaked in terms of its granularity and flexibility. There's already some flexibility in the current language (and I have seen many implementations that take advantage of this by not implementing certain keywords, such as unevaluated*, but still processing schemas declaring meta-schemas that require them). uniqueItems is an even better example because its memory requirements can easily become much larger than simply tracking what properties have been covered, and it is not in its own vocabulary.

Let's look at the language from §8.1.2:

Any vocabulary that is understood by the implementation MUST be processed in a manner consistent with the semantic definitions contained within the vocabulary.
...
The values of the object properties MUST be booleans. If the value is true, then implementations that do not recognize the vocabulary MUST refuse to process any schemas that declare this meta-schema with "$schema". If the value is false, implementations that do not recognize the vocabulary SHOULD proceed with processing such schemas. The value has no impact if the implementation understands the vocabulary.

While I did have in mind that implementations would refuse to process schemas if they didn't support all of the required vocabularies, that isn't precisely the language used here, which is about whether the vocabulary is understood. Various implementations have felt free to simply document that certain keywords are understood but not supported.

So, let's take our database implementation that can't support uniqueItems. It definitely understands the standard validation vocabulary. Even if I meant "understands" to be synonymous with "supports", we can adjust/clarify this (with an ADR now and a spec change before the next release) to be clear that "understands" just means "understands" and does not imply support. Then we just need to specify what happens with keywords that are understood but not supported.

We could clarify this by defining "understood" to mean that the implementation knows which keywords are part of the vocabulary, regardless of whether all of those keywords are supported or not.

§6.5 holds that unsupported keywords SHOULD be treated as annotations (and in the current text on main, MUST otherwise be ignored). We could update this language and that of §8.1.2 to cover the possible scenarios as follows:

• if it encounters an unsupported keyword from a known vocabulary present in $vocabulary with a value of true, it MUST refuse to [finish] process[ing] the schema (return an indeterminate/error result)
• if it encounters an unsupported keyword from a known vocabulary present in $vocabulary with a value of false, or a keyword from an unknown source (either an unrecognized vocabulary or a non-vocabulary extension keyword), it SHOULD treat the keyword as an annotation and MUST otherwise ignore it

Since folks often ask about the purpose of false here, I'll note that the false option is expected to be used for vocabularies like Hyper-Schema that do not impact validation, so that such a schema can still be used in a validator even if it does not understand Hyper-Schema (presumably, the hyper-schema keywords would be handled by a separate hyper-schema specific implementation after validation, if they're used at all).

This approach preserves the predictable fail-fast goals of $vocabulary. It won't fail quite as fast, but "fail-fast" here means "it fails before giving an unpredictable result, such as by silently ignoring a known but unsupported keyword."

[awwright]

That's not true. The value of $schema is only important in that it tells you how to get to $vocabulary, which has well-defined error behavior. It is only the lack of $schema that produces undefined behavior.

The issue is where a validator encounters a $schema value that it doesn't recognize, how is it supposed to handle this case? It could error, it could ignore it... No behavior is specified. Historically, $schema was strictly informational, it wasn't intended to change the semantics of the rest of the schema, and unknown values are still ignored.

---

It definitely understands the standard validation vocabulary. Even if I meant "understands" to be synonymous with "supports", we can adjust/clarify this (with an ADR now and a spec change before the next release) to be clear that "understands" just means "understands" and does not imply support.

Is there a meaningful distinction to the user/client? What do you get if a validator "understands" but does not "support" a feature?

The spec could just mention that implementations may implement a subset of validation features... but if a feature is unsupported, then you must not return a determinate, valid/invalid result.