Use OpenAPI to define MDS API request shapes and query parameters

[dyakovlev]

The existing documentation of query parameters and headers is in narrative format in the Provider and Agency README files. This is useful to give an overview of what's allowed, but because it is not programmatically enforced and left up to interpretation, it's common to see divergences from the spec. [1]

I'd like to propose adding API endpoint parameter definitions to Provider (and maybe Agency?) endpoints using a format designed to support automatic validation.

Implementers of MDS endpoints and software that interacts with them would be able to use these definitions to validate the shape of query parameters and headers for incoming requests. Additionally, a rigorous definition of the request shape would let MDS evolve to support more advanced request formats in a safe and transparent way.

A framework frequently used for this kind of schema is OpenAPI v3, an evolution of Swagger API definition formats. It has a lot of OSS and third party tools for API documentation and validation that may come in useful, and is reasonably well documented. Based on prior experience, I'd recommend using this format. [2]

---

[1] A similar thing to this already exists for validating the shape of responses via JSON Schema, and has been extremely useful for deciding if an API response conforms to the specification. However, JSON Schema is a toolkit for asserting the shape of JSON objects and does not really support defining the shape of request parameters and headers.

[2] I also considered suggesting JSON Hyper Schema (narrative intro), an extension to JSON Schema that builds a vocabulary for defining URI shapes. However, it is very young and does not have a really mature ecosystem of tools around it yet.

[dyakovlev]

notes from 5/23 call

• general consensus for need
• check that OpenAPI can support validating bulk data

some good thoughts about tool applications in the readme here - https://github.com/wework/json-schema-to-openapi-schema

also, having OpenAPI defs allows use of all of these tools which is pretty neat

[craastad]

I'm all for this, especially the choice of using the OpenAPI or JsonSchema tooling. I found the PHP tooling with JsonSchema to have issues with performance and completeness (i.e. adding massive overhead to requests to build responses and not not rendering nulls in paging params). I find the tooling to be better with OpenAPI. Please reach out if you'd like help with implementing or validating this change.

[hunterowens]

Any progress on this @dyakovlev @craastad ?

[dyakovlev]

not from my side, sorry

[marie-x]

Agree that OpenAPI is the way to go, for all of MDS (Provider, Agency, Policy, Audit ...), not just Provider, and agree that we should target this for 0.4.0.

[craastad]

I'm happy to look into this over the weekend.

[hunterowens]

sadly, it looks like the work integrating OpenAPI and JSONSchema is still a little incomplete (/ being led by a team at WeWork, so YMMV). I'm gonna assign this to the future milestone.

[schnuerle]

Do you all think OpenAPI is still the way to go and is a welcome feature to add now after we get 1.0.0 approved?

[marie-x]

I think that some sort of machine-readable API documentation would be a welcome addition. We should do a review to see if anything is better than OpenAPI.

[schnuerle]

There has been a call for something similar in CDS.

openmobilityfoundation/curb-data-specification#87

With MDS 2.0 the schema is going to need a major update. Typically we've relied on volunteers to complete this work each release, or even paid contractors to speed things up and double check work. I wonder if Swagger/OpenAPI could replace our json schema validation? And/or if we may need to fundraise the development of schema updates or OpenAPI creation in order to keep this for MDS 2.0 if there is not a volunteer or group willing to take on the work. I will bring it up with both WGSCs.

[marie-x]

I am in favor!

[schnuerle]

If time allows, we will bring this up for discussion at the public Working Group meeting tomorrow, so please comment here or attend and bring your thoughts.

[schnuerle]

We are getting more traction for OpenAPI in CDS.

JSON Schemas (as we've used already in MDS) are used for describing and validating data structures, which has been useful.

OpenAPI uses JSON Schemas internally as part of its data structure. Additionally it specifically documents the description (and sometimes validation) of HTTP requests parameters and responses for endpoints.

I think this addition of the endpoint documentation, definitions, and validation will be valuable for MDS consumers and producers and general users. I also think we can leverage most of the existing MDS JSON Schema work and incorporate it into the OpenAPI definitions. There are some tools to 'automate' this conversion for some syntax, notably this one made by Stoplight.

We will continue to work on this with CDS, and the idea there is to have a separate repo for the OpenAPI definitions. These will still be tied to release versions, but can be updated outside of the more formal spec release cycles. We will see how this plays out.

For MDS, the existing JSON Schema definitions would be removed from the spec itself and pulled out.

I also wanted to note that other specs like GBFS are still using JSON Schema, and I don't see talk of them moving to OpenAPI. Notably they put this schema in a separate repo, like I am proposing here.

[schnuerle]

We should be working soon on creating OpenAPI for MDS 2.0 in its own repo referenced by MDS.

[marie-x]

I strongly support OpenAPI for MDS 2.0, although I don't have the bandwidth to do it myself.

[schnuerle]

First draft of modifying MDS to support OpenAPI is happening with PR #832. This work to migrate to OpenAPI in a new repo should take about 2 months. More details coming soon.

[schnuerle]

The OpenAPI work is done for now with great work by @thekaveman!

See the end result here: https://github.com/openmobilityfoundation/mds-openapi

And visualized on Stoplight here: https://openmobilityfnd.stoplight.io/docs/mds-openapi/