OpenAPI validators

[pvdbosch]

In this discussion, you can share your ideas or experiences about validators that you use to verify if requests and responses comply with the OpenAPI specifications of your REST APIs that follow the REST guide.

[pvdbosch]

This thread is meant as a continuation of the discussions on OpenAPI validators in issue #25.

[pvdbosch]

I did some tests with the atlassian swagger-request-validator to check if it works well with the use of oneOf like we applied it in the BelgianRemittanceInformation schema

It seems to work as expected.

Error when both structured and unstructured fields are present:

{
"messages" : [ {
    "key" : "validation.request.body.schema.oneOf",
    "level" : "ERROR",
    "message" : "Instance failed to match exactly one schema (matched 2 out of 2)",
    "context" : {
      "requestPath" : "/submitRemittanceInformation",
      "apiRequestContentType" : "application/json",
      "location" : "REQUEST",
      "pointers" : {
        "instance" : "/",
        "schema" : "/components/schemas/BelgianRemittanceInformation"
      },
      "requestMethod" : "POST"
    }
  } ]
}

Error when none are present:

{
  "messages" : [ {
    "key" : "validation.request.body.schema.oneOf",
    "level" : "ERROR",
    "message" : "Instance failed to match exactly one schema (matched 0 out of 2)",
    "additionalInfo" : [ "/oneOf/0: Object has missing required properties ([\"unstructured\"])", "/oneOf/1: Object has missing required properties ([\"structured\"])" ],
    "nestedMessages" : [ {
      "key" : "validation.request.body.schema.required",
      "level" : "ERROR",
      "message" : "Object has missing required properties ([\"unstructured\"])",
      "context" : {
        "pointers" : {
          "instance" : "/",
          "schema" : "/oneOf/0"
        }
      }
    }, {
      "key" : "validation.request.body.schema.required",
      "level" : "ERROR",
      "message" : "Object has missing required properties ([\"structured\"])",
      "context" : {
        "pointers" : {
          "instance" : "/",
          "schema" : "/oneOf/1"
        }
      }
    } ],
    "context" : {
      "requestPath" : "/submitRemittanceInformation",
      "apiRequestContentType" : "application/json",
      "location" : "REQUEST",
      "pointers" : {
        "instance" : "/"
      },
      "requestMethod" : "POST"
    }
  } ]
}

[pvdbosch]

For the handling of unknown properties in JSON requests, like in rule-req-valid, I think it might be possible with the workaround of creating separate request and response validators:

var levelResolverResponse = LevelResolver.create().withLevel("validation.schema.additionalProperties", ValidationReport.Level.IGNORE).build();
var levelResolverRequest = LevelResolver.create().withLevel("validation.schema.additionalProperties", ValidationReport.Level.ERROR).build(); // this is actually the (non-spec compliant) default behavior of the validator 
var requestValidator = OpenApiInteractionValidator.createFor(openApiFile)
        .withLevelResolver(levelResolverRequest).build();
var responseValidator = OpenApiInteractionValidator.createFor(openApiFile)
        .withLevelResolver(levelResolverResponse).build();
        
requestValidator.validateRequest(...);
responseValidator.validateResponse(...);       

It might be more difficult to configure when using one of the built-in integrations of the validator like the spring one.

[JDMKSZ]

In the search for having a validator that validates 1) all examples and 2) that all security scopes are defined in the securitySchemes, I found that

1. Only openapi-examples-validator has the option to refuse all properties that are not defined in the schema (-n). It also has the option to require all properties defined in the schema to be present in the example (-r). Typically we don't need the -r, because you'd want to create examples with not all properties present (e.g. an example of a resource with "minimal" info present and one with "maximal" info present, or with info A present and one with info B present). However, we DO want the -n, to avoid examples becoming outdated when renaming properties (in the API design phase). Usage: npx openapi-examples-validator -n src/main/openapi/openapi.yaml. Note that you need to ignore the numerous "additionalProperties" flag not set warnings, see here why.
2. Only Spectral reports scopes that aren't defined in securitySchemes. The spec says says "In case of OAuth 2, the scopes used in security must be previously defined in securitySchemes.". Spectral has such a validation rule in its default ruleset. Usage: echo extends: ["spectral:oas", "spectral:asyncapi"] > .spectral.yaml; npx @stoplight/spectral-cli lint src/main/openapi/openapi.yaml --ruleset .spectral.yaml

These are the tools that I tried:

1. Redocly CLI
2. Smals/Belgif styleguide validation plugin 1.4 -> examples are only validated w.r.t. the schema, but not the presence of undocumented properties
3. openapi-style-validator
4. openapi-examples-validator
5. Spectral
6. openapi-linter by superfaceai

One more that I found but did not evaluate: IBM openapi-validator

[pvdbosch]

I opened issues on the belgif plugin project for this functionality:

• Verify scopes are defined in security scheme rest-guide-validator#3
• Validate that example values don't have undefined properties rest-guide-validator#4