Conformance tests

[scothis]

We have a spec that has requirements for implementations. It would be nice to have a test suite that could be used to judge if an implementation is compatible with the specification.

The conformance tests should:

• have an assertion to cover each MUST, MUST NOT, etc in the spec that will fail the suite if the assertion fails
• have an assertion to cover each SHOULD, SHOULD NOT, etc in the spec, but do not fail the suite if the assertion fails
• have a negative test to cover each MAY, RECOMMENDED, etc in the spec, but do not fail the suite if the assertion fails
• make no assumptions as to ambiguities in the spec (those should be files as issues against the spec)
• make no assumptions as to the internals of the implementation, only the spec defined interfaces should be tested
• have minimal prerequisites (ideally none) other than a service binding implementation running in a cluster
• remove any state they add to the cluster during the tests

Initially, we should focus on the core specification. Spec extensions should discrete conformance tests that can be run independently.

[arthurdm]

great idea @scothis - it will be crucial for ensuring adopting frameworks remain consistent.

[arthurdm]

Community to check whether there's guidance or requirements from a SIG perspective on conformance test suites.