Validate HTTP responses against example values while running tests

[harikrishnan83]

We have received several queries about why Specmatic contract tests only verify the schema of HTTP responses based on OpenAPI specs? Why do we not verify the values in these responses agains the examples in the OpenAPI spec or externalised test data JSONs.

Our response has been that verifying the schema of response validates the signature / contract of the API, however validating the value will result in Specmatic contract tests also verifying application logic. We have been trying to separate these two concerns (Contract vs API / Component tests) so far.

However as an experiment we have added below env variable as a flag to turn on response validation (default false).

VALIDATE_RESPONSE=trueVALIDATE_RESPONSE_VALUE=true

We request your valuable feedback on this capability (available from release 1.3.1 onwards).

UPDATE: The name of the env variable above has been updated as of release 1.3.13

[hernaneg350]

Some of us run contract tests against an application with mocked dependencies where the only real part involved is the Controller layer (as in MVC). For these types of tests we are able to provide mock responses, where we can validate how the controller transforms them to JSON beyond the scope of syntax which schema validation provides (by doing this, we focus tests on the contract side, without verifying application logic, which run faster, and allow us to focus the rest of the testing efforts on unit tests, integration, and E2E or component).

In addition, and this is not less important, using the actual examples in the Swagger for validation allows us to validate the documentation as well. Documentation is never going to base itself on examples that are wrong.

[harikrishnan83]

@hernaneg350 Thanks for contributing to this thread. +1 on having validated examples in the OpenAPI spec thereby making for accurate documentation. That is indeed really important.

About testing only Controller in ContractTest, IMHO depends on how we are categorising the ContractTest and where it appears in the TestPyramid. So far we have been running ContractTest in a manner similar to Component Test (System Under Test is the entire MicroService, External dependencies such as DB, Messages Queues, Other HTTP services, etc. are mocked). Here is video from YOW! conference detailing the anatomy of a Contract Test setup. This video goes into effectively isolating the SUT from its dependencies.

Even this setup has been consistently fast, given that all external dependencies are mocked. Please have a look and we would love to hear you feedback.

UPDATE: Also if the application is architected well, with appropriate layering, then it may be possible to ContractTest the controller. However quite often while dealing with legacy application it may not be possible to expect DI, layering in order to isolate the Controller by mocking other layers.

[hernaneg350]

This is great @harikrishnan83, will definitely look into it and get back to you.

[hernaneg350]

Hi @harikrishnan83, I looked at the video and from what you said I am leaning towards two conclusions:

First, indeed testing the controller might not always be possible. I started thinking that from a certain theoretical point of view, a protocol contract is where an MVC controller opperates, so in theory testing the contract could be seen as testing how the controller translates instructions in the language of the contract to domain instructions that the domain layer understands. I have come to the conclusion that such a purist approach is usually harder to implement and maintain in spirit than just creating Component Tests that strategically isolate the SUT into an appropriate layer for the use case. I agree with you about this.

Second, I have also discovered great value in being flexible about the notion of what a contract is. This means that a contract may not necessarily or desirably be strictly defined by an OpenAPI file and its request/response type schema, when planning out an automated testing strategy. Even though theoretically the compliance with an OpenAPI definition is the only thing I should take as guaranteed when dealing with an API, in practice the contract informally extends more guarantees. An example of this is expecting that a GET /user/1234 after a PUT /user/1234 should not result in a 404. There is no part of an OpenAPI definition that can specify this, yet it is somehow still part of a contract I would not want to break (a real world example of me breaking this is switching from a strongly consistent store into an eventually consistent one that does not support Read-Your-Writes). There is a certain balance here and it is not science, I certainly cannot support every bizarre use case that my clients pretend, and guarantee never breaking, yet my clients will definitely need more guarantees than just an API definition file. Consumer-driven contract testing has long understood this problem in letting certain consumers drive the compliance tests by specifying their use-cases, yet these are not suitable for all cases, specially when the provider needs to keep full control of these guarantees and their release cycle (a very typical scenario when you take into account vendoring).

I think this is where the magic of Specmatic comes in, but it can only do so if it merges more and more with the world of traditional API/component tests in providing tools to test guarantees that exceed format. I think VALIDATE_RESPONSE_VALUE is indeed a step in the right direction, and I have been enjoying the feature enormously in one of my new services. It lets me do TDD against my API with minimal coding (just the Specmatic test files), levereaging auto-generated cases and data, testing against the OpenAPI definition, and including coverage information, all in one. This is true provider-driven contract-testing for me.

May I suggest, in light of this, more of these types of features:

• Validating headers and other parts of the response (my new API test suite would be perfect with this, as I would not have to code a single component or controller test with JUnit).
• Validating derived values, when a value in the response is derived from a value in the request (very simple example: returning the ID of a new entity, where the ID is present in the URL).
• Merging validation of OpenAPI examples with validation of external example files: Not every test case can be declared via OpenAPI examples, yet I would want to keep testing the examples if some of my cases have to be in separate files.

Let me know if I can provide my help in any feature request, I will be happy to do so!

[harikrishnan83]

@hernaneg350 Thanks a lot for taking a look at the video and also sharing your detailed response.

We sincerely appreciate the time and thought you have put into this.

We will go through your suggestions in detail and get back to you on the same.

[kiranpatel11]

Thanks @harikrishnan83 for this much awaited improvement.

my thoughts on the below

Our response has been that verifying the schema of response validates the signature / contract of the API, however validating the value will result in Specmatic contract tests also verifying application logic. We have been trying to separate these two concerns (Contract vs API / Component tests) so far.

Yes it is true that contract validation should not care for specific value, but the purpose of this capability SHOULD be to validate the "shape-of-the-payload" (but it is not atm), meaning the expected combination of the keys are present in the response for a given request.

I mentioned the usecase in this thread : #709

[yogeshnikam671]

Hey @kiranpatel11, just to clarify, the purpose of this capability is actually to validate the specific value being returned in the response against the example data provided in the specification for the response in addition to validating "shape-of-the payload".
We will make sure, we articulate this clearly in our documentation when we make this feature more widely available.
Thanks for your feedback!

[kiranpatel11]

@harikrishnan83 , I tested with the above flag, but unfortunately it is not helpful.

As I mentioned, the contract test should not be validating the specific value, otherwise it will be flaky tests. However, my expectation is to validate response to ensure all JSON keys (including the optional) match with the expectation.

[kiranpatel11]

@harikrishnan83

1. In my testing, I have not seen specmatic validating the "shape of the payload" for response as defined in the example. Hence I raised Real world challenges with contract testing and integration issues #709 sometime back.

Here is another simple example, the field visitRequired is conditionally required field (see the business rule in the description field of the schema). When my implementation is not returning that field for the petId 9991, I expect the specmatic test to fail, but it passes !! and not validating the response as per my defined expectations, but only validating it against the schema !!!

I expect the introduction of the VALIDATE_RESPONSE=true to solve this problem, but rather it introduced the another problem -- to validate exact values, which is not a best practice for the contract testing in most cases, baring few exceptions like enums.

  /pets/{petId}:
    get:
      summary: Get a pet by ID
      parameters:
        - name: petId
          in: path
          required: true
          schema:
            type: integer
          examples:
            GET_PET_SUCCESS:
              value: 9991
      responses:
        '200':
          description: Success - Pet found
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/Pet"
              examples:
                GET_PET_SUCCESS:
                  value:
                    id: 9991
                    name: "Fifer1"
                    birthDate: "2020-12-24"
                    owner: "John"
                    visitRequired: true

components:
  schemas:
    Pet:
      type: object
      properties:
        id:
          type: integer
          readOnly: true
          example: 9991
        name:
          type: string
          example: "Fifer"
        birthDate:
          type: string
          example: "2020-12-17"
          format: date
        owner:
          type: string
          example: "John"
        visitRequired:
          type: boolean
          description: true if the pet visit is due (> 6months from last visit) or false otherwise
          example: true
      required:
        - name
        - birthDate
        - owner

I am happy to have a conversation rather than these long responses.

[kiranpatel11]

2. on the "mis-spelt optional field in the response" - If the implementation returns field not defined in the schema, shouldn't it get detected via the schema validation ? am I missing anything ?

[harikrishnan83]

@kiranpatel11 Yes, we can certainly discuss over a call. And as always we appreciate your valuable feedback. Please reach us through our contact us page and we will setup a call with you. Thanks.

[kiranpatel11]

@harikrishnan83, @yogeshnikam671

The above mentioned problem (/pets api example) is resolved with 1.3.13. It does not require the new flag VALIDATE_RESPONSE_VALUE either for this particular problem.

The usecase for VALIDATE_RESPONSE_VALUE is now different from the problem I raised, and now better naming too. As long as it remains optional, I don't have issues. As you origionally mentioned, the value validation is not really in the remit of the contract testing (baring few exceptions), I suggest you clarify the usage in the documentation and its trade-offs.

[harikrishnan83]

@kiranpatel11 Glad you noticed :). Yes, we have been working on some of these changes. Should soon reflect in our documentation.

[gurubabujampala]

Hi @harikrishnan83, I can't find the version 1.3.13 in maven central repository.

[harikrishnan83]

hello @gurubabujampala, Thanks a for reaching out.

I see that 1.3.13 is available in Maven Central - https://repo1.maven.org/maven2/in/specmatic/specmatic-executable/1.3.13/.

Sometimes we have noticed that this page takes time to reflect - https://mvnrepository.com/artifact/in.specmatic/specmatic-core.

Please update your pom.xml, build.gradle files with 1.3.13 and it should work. We have tested for the same on Github Actions also and it is working.

[gurubabujampala]

@harikrishnan83 sure will check, thanks for your quick response.

[hernaneg350]

I couldn't help but notice that the flag does not act on externalized test data Quite a problem since we have no way of specifying header example values inside the OpenAPI spec (at least specmatic will ignore header examples in the request).

Also noticed that examples in OpenAPI file are "either/or", so I either run the examples on the OpenAPI file or test against JSON files in the tests folder.

Am I correct in my diagnosis or might I be missing something?

[barbeque-squared]

@harikrishnan83 I have a question (bug?) about this (otherwise really awesome, thanks for the hard work!) new feature when an endpoint doesn't have any parameters. See the attached zip for a minimal example to reproduce: specmatic-no-validation.zip

The endpoint / takes no parameters (you can comment out the entire parameters section in the yaml), but then it also doesn't do any response validation. As soon as I add a dummy parameter with a completely irrelevant expamle value, it does start doing response validation.

As a workaround I'm currently just programmatically adding these dummy examples to requests without parameters, since I'm going to guess it has something to do with success linking the request to the response?

The one edge case that I can think of, is if there's more than 1 example for a given status code. But I don't see how that would be useful for automatic specmatic validation anyway, so my suggestion would be to throw an error if there's multiple examples for a parameterless request and VALIDATE_RESPONSE_VALUE=true. And if there's exactly one example, use that for validation purposes? If not, this should probably get documented somewhere, because for me it was very non-intuitive that my requests with parameters were getting validated, but not the ones without.

[harikrishnan83]

@barbeque-squared apologies for the delayed response. That is an interesting scenario. I will try out the sample you have shared and get back shortly. Thanks again for the details.