Real world challenges with contract testing and integration issues

[kiranpatel11]

Many real-world APIs are defined using the open-ended schema (like key-value pairs) for the genuine reasons.

e.g.

• TMF Open APIs used as standard for Communication Service Providers.
• eCommerce APIs like ProductCatalog, Cart or Order which are driven by the catalog-driven content - business rules, characteristics, etc.

I know the schema of such dynamic content is still important, without that things are not expected to work e2e (see the dynamic schema references in TMF APIs). However, in realworld they are not used consistently unfornately and not much tooling around it.
Usually the behavior of such APIs are expected to be defined using the BDD style requirements. However, using the Specmatic schema-based approach, the consumer stubs validation against the schema, does not add any additional value as stubs are not getting validated against the allowed/possible behavior described by the provider.

Consider the below OpenAPI specs for the detailed explaination.

openapi: 3.0.0
info:
  title: Order API
  version: 1.0.0
paths:
  /orders:
    post:
      summary: Create a new order
      requestBody:
        required: true
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/Order'
      responses:
        '201':
          description: Order created successfully
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Order'
        '400':
          description: Bad request - Invalid input data
        '500':
          description: Internal server error
components:
  schemas:
    OrderItemCharacteristic:
      type: object
      properties:
        key:
          type: string
        value:
          type: string

    OrderItem:
      type: object
      properties:
        id:
          type: integer
          format: int64
        product:
          type: string
        quantity:
          type: integer
          minimum: 1
        characteristics:
          type: array
          items:
            $ref: '#/components/schemas/OrderItemCharacteristic'
      required:
        - product
        - quantity

    Order:
      type: object
      properties:
        id:
          type: integer
          format: int64
        customerName:
          type: string
        items:
          type: array
          items:
            $ref: '#/components/schemas/OrderItem'
      required:
        - customerName
        - items

Just to exaggerate, the provider may have defined behavior to accept IPhone, Broadband products order, but consumer create stubs using the grocery items but valid against the schema for verifying their implementation.

Hence, IMO, the behavioral aspects are important in Service Virtualization, and Specmatic comparision with Pact and PactFlow ignore such usecases and appears to be mis-leading that all realworld integration issues can be resolved using the schema-based validations.

IMO, this problem can only be solved by "subset compatibility" checks of stubs with provider behavioral scenarios.

[harikrishnan83]

@kiranpatel11 thanks for starting this thread.

Before I address the main point of this topic, just want to clarify that we are not claiming anywhere that "all realworld integration issues can be resolved using the schema-based validations." in our comparison at all. In our point "Minimal / No Leakage for Compatibility Issues" we are clearly calling out that we catch issues where there is deviation from the API specifications. When the specification only captures structure / signature of the API and not the behaviour, we can only validate to that extent. Not sure what makes you think this way. Can you please elaborate?

[kiranpatel11]

@harikrishnan83, on the specmatic website many direct or indirect claims are there such as

• "Shift left and kill integration tests"
• Independent Deployment
• identify compatibility issues

I don't want to sound negative or critical about the specmatic. I truely believe it has potential, but it requires lot more improvements to achieve such claims.

As mentioned by the veteran, Dave Farley, in the video reference on your website, pact has been recommended tool for contract testing and remove the need of e2e integrated testing in connected environment so far. Even they have written a lot about the schema-based vs test-based contract testing. If specmatic aims to replace Pact, then as a user, I expect it to be honest in what aspects of the contract testing and integration issues it still does not resolve. On the consumer side, validation stubs against schema gives little benefit (necessary but not sufficient) from removing the need of integration testing for decoupled microservices, particularly if the schema is open-ended. I would expect some blog post/documentation on how specmatic deal with such real-world usecases, and recommendations or oppiniated solution address the situations.

[kiranpatel11]

Technically, I know one way to solve the above usecase.
The OrderItemCharacteristic is a key-value pair (open-ended), but such keys should still be part of the provider contract, usually not hardcoded/static in the OpenApi specs, but either via another documentation (prone-to-errors) or dynamic schema.

e.g. This is an example of the ProductSpecification from the TMF APIs, mentioned above.

This is particularly useful in a chain of services, where many of the services act as pass-through orchestrator for subset of the payload.
In the above example, the usually the services won't have hardcoded business logic against the "OrderItemCharacteristics" object, but many in the chain have to act as a carrier/pass-through of such dynamic data. Having the dynamic schema (usually available in the central schema registry) helps such services validate the payloads in an abstract way without having to be aware about such fields individually.

The reason I am mentioning this here, is for the opportunity for specmatic to support validating the stubs against such dynamic schema. Reference : https://json-schema.org/blog/posts/dynamicref-and-generics

[kiranpatel11]

Question for you @harikrishnan83,

Considering my orderItem.characteristics example above for open-ended schema, lets say I still have detailed scenarios captured using the specmatic DSL (.spec file) about the possible combinations of characteristics for a given product. When I use such .spec file for stubbing on the consumer side, would it be able to verify that the stubs matches the defined combinations for a given product ?

I guess it will not, because the schema does not really exist, but thought of confirming with you.

[kiranpatel11]

extending the above question for the provider too.

When the example for a particular response involve the optional fields, shouldn't the specmatic consider it for the expected assertion ?
e.g.

Given the below OAS, where I am trying to model the different "shape of the response payload" to cover the behavior variation.

  /pets/{petId}:
    get:
      summary: Get a pet by ID
      parameters:
        - name: petId
          in: path
          required: true
          schema:
            type: integer
          examples:
            GET_PET_SUCCESS_1:
              value: 1
            GET_PET_SUCCESS_2:
              value: 2
      responses:
        '200':
          description: Success - Pet found
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/Pet"
              examples:
                GET_PET_SUCCESS_1:
                  value:
                    id: 1
                    name: "Fifer1"
                    birthDate: "2023-09-07"
                    owner: "John"
                GET_PET_SUCCESS_2:
                  value:
                    id: 2
                    name: "Fidler"
                    birthDate: "2023-01-07"
                    owner: "Jack"
                    visitRequired: true

(My) Expectation :
When the field is explicitly included in the response example, I expect the tool to assert its existence.

AS-IS behaviour:
When the field is explicitly included in the response example, but the implementation does not return the optional field, the test PASSES as it is still valid as per the schema.

[kiranpatel11]

related discussion #1029 , though not verified yet.