Proposal: support fragment identifier in path object

[link89]

OpenAPI spec is design for modeling REST APIs, but in real world we may find some APIs are mixture of restful style and RPC style.
Those RPC APIs may share same URI, but their schema or content of their request body or query parameters are different.
It's common to meet this problem when you try to model APIs that have been in your company for a long time.
As far as I know OpenAPI spec cannot model those kind of APIs properly because the combination of verb+path uniquely identify an operation.
I found that there are other people have similar problem with us:

swagger-api/swagger-core#935
swagger-api/swagger-editor#854

But we found that this problem can be solved by simply adding fragment identifier in path objects.

Here is the example. There are some RPC style APIs sharing same path /api/team/{team_id} and verb "POST". Their request body may look like

// Add members
{"operation": "add members", "data": {"members": [1, 2]}}

// Remove members
{"operation": "remove members", "data": {"members": [3, 4]}}

(I know they are bad designed, but they do exist and we have to model them.) By default, tools like Swagger-UI/Editor or swagger code generators cannot handle this case properly because their path and verb are same. In order to model those APIs, we add a fragment identifier after path for each RPC APIs

/api/team/{team_id}#AddMember:
    post:
        ...
/api/team/{team_id}#RemoveMember:
    post:
        ...

And Swagger-UI/Editor will treat them as different paths after we add this fragment, so that we can now model these RPC APIs like other restful ones in swagger editor.
Because standard HTTP clients (browsers, ajax, urllib/requests in python, etc) will automatically omit fragment in url when making reqeusts, even we add fragment in path object, tools like Swagger-UI/Editor or clients created from swagger code genenerators can still work properly.
And that's all we need to do.

Accoding to wikiepdia

In computer hypertext, a fragment identifier is a short string of characters that refers to a resource that is subordinate to another, primary resource. The primary resource is identified by a Uniform Resource Identifier (URI), and the fragment identifier points to the subordinate resources

I think it is reasonable to use fragment identifier this way in OpenAPI.
And I hope this proposal can become parts of OpenAPI spec to help other people who have similar problems.

[silkentrance]

The fragment identifier, correct me if I am wrong, is something that gets evaluated on the client side and not on the server.

In the past, fragment identifiers have been used to maintain a client side only history and application state, and specific code needs to be in place in order to handle such.

The server does not know about such fragment identifiers, see for example here: https://en.wikipedia.org/wiki/Fragment_identifier

[...]
Fragments depend on the document MIME type and are evaluated by the client (Web browser). 
Clients are not supposed to send URI-fragments to servers when they retrieve a document, and 
without help from a local application (see below) fragments do not participate in HTTP redirections.[...]

So in order to be able to evaluate fragment identifiers on the server, your client side application will have to make them actual parameters of either REST call, e.g. parameter: fragment { type: string }

[link89]

@silkentrance You are right. I don't take server side into account because we only used code generators on client side. I do know that fragment identifiers only work on client side. The point here is we find that we can use fragment identifiers to further distinguish difference resources under same paths, by doing this we could have independent entries for those sub-resources in auto-generated clients.
And since fragment identifier won't be send to server side, the code generators and swagger-ui/editor can still work properly even we add them at the end of paths.
By introducing fragment identifier we can simply solve the problems like swagger-api/swagger-core#935. Fragment identifier make those kind of resources look like different paths on client side, but those requests will be sent to same endpoints on server side, HTTP clients will omit fragments automatically when making requests.

[iongion]

This would be so useful - especially in Role based Service APIs, where what OpenAPI considers a service identifier path + verb can have a polymorphic spec(the entire spec can differ), depending on the Role, without violating any considerations. Is just as if the API transforms itself depending on some domain specific switching criteria.

PS: Of course Role has no meaning in OpenAPI , but it is a common switch criteria of service behavior in current or legacy API designs.

Examples:

• Registered users in a user system api have different /GET profile than non-registered, non-confirmed ones.
• Listing books in a book management systems allow more input filters for library managers and respond differently than for library users.

[darrelmiller]

This is both horrendous and ingenious at the same time. This would allow folks to model RPC style APIs where a single path tunnels multiple operations. However, we probably would need some way to correlate the value in the fragment with some value in the request body. Or at least support the notion of constant parameter values that allow the server to distinguish between the different operations.

It does raise the question, do we allow this use of fragment identifiers and twist the definition of paths or do we add semantics to natively support this HTTP usage pattern.

[handrews]

However, we probably would need some way to correlate the value in the fragment with some value in the request body.

Are we talking about Link Objects here? Or Operations? It's pretty common for there to be a correlation between request body values and path or query string parameters, but we don't really describe that in Operations today.

Or at least support the notion of constant parameter values that allow the server to distinguish between the different operations.

ahem... modern JSON Schema const 😁

[link89]

ahem... modern JSON Schema const 😁

const is a great idea for the case that using a enumerated field in request body to distinguish the operations. But for more complex cases I think we need a Turing complete JSON precondition DSL to decide which backend service should response to the request. I don't think there are exited tech can meet this requirement.

IMHO, a more practical method is for endpoints that share same path, the server-side code generator should force the backend developer to implement a precondition(request)->boolean interface. And backend router should use both path pattern and result of precondition to decide what controller the request should be routed to.

[ashish25cca]

@link89, and others:

I'm stuck with the same problem.

I've one base path-URL and all APIs are listed under this, APIs have different query parameters.
So, to resolve this issue, I used a fragment identifier for each API. Now, Swagger is not giving me any error and I'm receiving API response as well.

But, the response is not correct because the fragment identifier is attached to the request URL and curl request.

openapi: 3.0.0
info:
title: API Documentation
description: >
This documentation gives you real time experience APIs you can interact with.
servers:
url: https://test.server.in/merch
description: Optional server description, e.g. Main (production) server
url: https://secure.server.in/merch
description: Optional server description, e.g. Internal staging server for testing

paths:
/postservice#api_to_vefiry:

**post:**
  summary: API to develop a program by providing all the building blocks.**

Now, the fragment identifier is attached with the request URL:

https://test.server.in/merch/postservice#api_to_vefiry?form=2


and, the fragment identifier is also attached with curl request:

curl -X POST "https://test.server.in/merch/postservice#api_to_vefiry?form=2" -H "accept: application/json" -H "Content-Type: application/x-www-form-urlencoded" -d "key=gtKFFx&command=verify_payment&var1=20190729&hash=063457c2c37daed990c8a6aee86a993df72dcd48ddef8fa384e587222a12529a2549d74b3713cb721d49170d5ac73e8835c0b2ca08a39323af8747becb8368bf"

The issue can be resolve:
1). if fragment identifier is not attached in request, so the response will be okay.
2). else, is there any other way to achieve the same functionality. e.g. API listings under same base URL. Please note each API has its own different query parameters.

Thanks in advance :)

[link89]

@ashish25cca You need to modify the codegen a little bit to workaround this problem.

Most of swagger code generators do not handle fragments properly. They just concat query-string right after the path instead of moving fragment to the end of query-string. You can fix this in codegen by simply remove fragment before join with query-string. Or more formally, using URI class (which is part of stdlib of most programming languages and will handle this case correctly) instead of string operation.

The problem you have occurred is the main motive I create this PR. Because the usage of fragment identifier is not covered by OpenAPI specification, most of codegen implementation don't take this into account (actually they should because it's covered by URI spec). But it's easy to workaround (with one or two lines of code changed), so I think you can try to fix it this way.

[ashish25cca]

@link89 : Thank you so much for your quick reply.

Would be a great help if you kindly provide the steps to fix this in codegen?

[ashish25cca]

@link89 : I've downloaded swagger editor from below link and writing yaml for each API manually.

https://github.com/swagger-api/swagger-editor

If you can please let me know the file name with the required change?

,and thank you so much for your time : )

[link89]

@ashish25cca

https://github.com/swagger-api/swagger-js/blob/fe38245efdde02f0508a6b6a4f0fb59215d2e924/src/http.js#L263

I guess you need to change req.url = baseUrl + finalStr to req.url = baseUrl.split('#')[0] + finalStr and reinstall swagger-client with the modified one.

I didn't test this by myself, you may give it a try and give me some feedback.

Update:
a quick and dirty work around:
open file swagger-editor$ vim node_modules/swagger-client/dist/index.js and navigate to around line 741 and change the code.
Then execute npm run dev to start a dev server, it just works.

[ashish25cca]

@link89 , thank you for your reply.

However, I've tried changing code in node_modules/swagger-client/dist/index.js at line number 740 with below code:

var finalStr = joinSearch(newStr, encodeFormOrQuery(query));

req.url = baseUrl.split('#')[0] + finalStr
delete req.query;

}
return req;


But, no luck : ( still segment identifier is being added to request URL and curl request., please suggest further. Is there something I'm missing out.

[ashish25cca]

@link89, hey, I'm seriously missing something. So, this is I've done:

1). Downloaded swagger editor from this link:
https://github.com/swagger-api/swagger-editor

2). Now, since node_modules are not present into this by default, I went to the director location through terminal and run

npm run dev


3). So, by now I have node_modules folder in my project, I changed code in node_modules/swagger-client/dist/index.js at line number 740

var finalStr = joinSearch(newStr, encodeFormOrQuery(query));

req.url = baseUrl.split('#')[0] + finalStr

delete req.query;

} return req;

4). When I run it through local host, still fragment are not removed:

Kindly suggest, would be a great help !!

[link89]

I guess you can try to rerun npm run dev and using incognite window (in case of cache) to open it and try again

[ashish25cca]

@link89, I've tried all but no luck, still, fragment is attached to request URL.
Please provide the steps in details from downloading link to final execution, you have run the project.

It would be really great help.

[ashish25cca]

@link89 , Thank you so much for your help so far :)

@darrelmiller , @iongion ,@silkentrance : guys I need help..
Kindly provide me steps you have followed to run it, I'm missing something here.

Please provide video for the same, if possible.

[link89]

@ashish25cca I guess you need to learn to fix this by yourself, for example adding some console.log or call debugger and tunning the code in your browser.

[AshishPayU]

@link89, apologies to update on it so late.
Yes, I was able to run it the next day to it. Thank you so much for this, really appreciate your help :)

[giovinazzo-kevin]

+1.

We're designing a contextual API that changes spec depending on the current tenant, which is specified by a request header value. So the same endpoint might actually perform two different operations depending on this value.

Every controller is thus mapped as such:

/api/{tenant}/[controller] (explicit routing)

/api/[controller] (implicit routing)

This issue is a deal-breaker because the frontend team relies on tools that generate Typescript controllers from a Swagger/OpenAPI spec. If we enable implicit routing the spec breaks, because now there are multiple controllers that point to the same path despite only one being able to answer the client at a time.

For the time being, I'm either going to disable implicit routing when generating the spec or I'm going to write some fancy document processor for NSwag that ignores implicit routes. But it would be great if OpenAPI implemented some way of dealing with such ambiguities.

[handrews]

@darrelmiller

It does raise the question, do we allow this use of fragment identifiers and twist the definition of paths or do we add semantics to natively support this HTTP usage pattern.

Personally I'd advocate for natively supporting the pattern if it is considered in-scope for OpenAPI at all. URI Fragments have very well-defined behavior: They are purely client-side, and the exact fragment syntax and semantics are determined by the resource representation's media type, not the protocol indicated by the URI scheme.

No library that correctly makes HTTP requests will include the fragment in the request-URI.

The implicit routing use cases here appear to be:

• The routing is done based on a header value
• The routing is done based on query parameters
  • This may involve just one or possibly several parameters and values
• The routing is done based on the request body
  • In simple cases, there is an enumerated field that determines this
  • However, the body-matching can be arbitrarily complex

OAS describes headers, query parameters, and request bodies in different places, all separate from the path. Fragments are not addressed at all, as OAS is concerned with client-server communication, while fragments are part of media type specifications.

---

Given that this issue and its corresponding PR are not on @earth2marsh 's list of 3.0.3 candidates (#2130), I'm guessing this would be handled in 3.1 at the earliest. It's not clear to me that even that "tack on a fragment" solution is sufficiently compatible to be put into 3.0.x at all (can anyone confirm that one way or the other?)

In OAS 3.1 we could solve this by adding a new OAS-specific schema extension keyword that could be used to correlate across all of these things. You would use a oneOf in the schema of each thing that needs to correlate, possibly with an if/then in each branch if you want to explicitly call out the predicate for the server to test.

The extension keyword would be used in each oneOf branch to name it. Since that keyword would indicate that you are correlating things, an if/then alongside of the keyword would be taken as expressing the correlation predicate (there are other reasons you might use an if/then so it's useful to disambiguate the desired behavior).

If you want to forbid certain parameters or headers in a particular branch, you can set their schemas to false (which is the same as "not": {} but with clearer intent).

That's all pretty hand-wavey. I can work up an example if there is interest.

Of course we could also solve this by restructuring the Path Item or Operation Objects, but that would have to wait until OAS 4.

[handrews]

Note that this approach could also be used for other sorts of correlations, such as correlating Prefer header values in a request with Preference-Applied header values and/or representation structures in responses.

[earth2marsh]

FWIW, I'm bearish on the idea that fragments have meaning here. If you consider that a spec is not a client-only artifact, such as using the spec to generate server-side code, then fragments are meaningless, since they're purely client-side. Happy to discuss and be enlightened, but I don't see that fragments belong in the spec, myself.

[DavidBiesack]

See also #182 for how fragments can (be abused) to overcome OAS constraints. They need not be part of the actual routing/dispatch at runtime; the server can multiplex requests based on actual parameters. As Darrell says, horrendous and ingenious at the same time

[jaypipes]

FWIW, I'm bearish on the idea that fragments have meaning here. If you consider that a spec is not a client-only artifact, such as using the spec to generate server-side code, then fragments are meaningless, since they're purely client-side. Happy to discuss and be enlightened, but I don't see that fragments belong in the spec, myself.

How do you propose that separate API operations be differentiated in the spec, then, if the Path and HTTP Method are the same?

For reference, I've been trying to model AWS APIs like SNS or SQS using unmodified OpenAPI3 Schema and run into issues because every single API operation is POST / with query parameter values for action differentiating what the operation is :(

I know it's a terrible design for APIs, but it's not like AWS APIs are going to fundamentally change any time soon and it's a shame that really just due to this issue I cannot properly use OpenAPI3 Schema to model these types of APIs without the fragment hack..

[MikeRalphson]

@jaypipes FWIW we use the fragment hack over at APIs-guru in documents such as https://github.com/APIs-guru/openapi-directory/blob/master/APIs/amazonaws.com/sns/2010-03-31/openapi.yaml

[link89]

@earth2marsh

If you consider that a spec is not a client-only artifact, such as using the spec to generate server-side code, then fragments are meaningless, since they're purely client-side.

That's not true. In essence, swagger is just a formal document. Users can parse the document and generate code from it, all you need is to handle fragment (use with Specification Extensions ) in path objects properly if you want to use it to generate server-side code.

And I believe most of the users won't have these problems because if you are seeking a solution to handle the problem discussed in this thread, then I can bet your server-side code is not generated from swagger document 😂

From my experience, swagger is most useful as an API modeling/documentation tools and client-side codegens. By introducing fragment into path objects won't break these features. And it also compatible with server-side codegen, all you need is to customize the codegen to handle these special cases.

Cannnot model APIs whose path and HTTP method are the same is one of the deficiencies that cannot be workaround from external but to fix the OAS specification itself. And the beauty of fragment solution is that it is totally compatible with the current specification. If you don't use it, it won't break anything. If you need to use it, it works well with swagger editor and most client side codegens.

We are looking forward to the new features introduced by OAS4. But before we reach that point I think it can be no harm by introducing this hack. It just works well to use with YAML templating tool ytt to reduce unnecessary redundancy code in our use case.

[handrews]

That's not true. In essence, swagger is just a formal document. Users can parse the document and generate code from it, all you need is to handle fragment (use with Specification Extensions ) in path objects properly if you want to use it to generate server-side code.

The server can't see what fragment was used in the request. HTTP does not allow it.

[earth2marsh]

Even though I disagree with the fragment identifier, I do support exploring alternate signatures. If several API operations share an HTTP method and path pattern but can be identified by an attribute in the query string or payload, there might be other was to represent that. However, it is worth noting that this expansion would add some additional burden on tooling authors who would have a bigger surface area to support.

[darrelmiller]

Query parameters are part of the resource identifier. We need to get to a point where we can distinguish between PathItems using the query parameter. Probably one of the biggest stumbling blocks is the property name "paths". Once we get over that, we then have to deal with funky stuff like the significance of parameter order. Most people don't consider the query parameter order as significant, but as far as resource matching is concerned it is.

This doesn't solve the problem of distinguishing operations by headers and request body content. But I think we need to solve this problem one bite at a time.

[link89]

@handrews

The server can't see what fragment was used in the request. HTTP does not allow it.

Sure I know this and that's why I said to use with Specification Extensions for server-side code-gen. For example, if there are two operations, add members and delete members, under the same path /api/teams/{teamId}, whose routes are decided by a field in request body named action. Suppose the server-side code is written with typescript, then we can model it this way

/api/teams/{teamId}#action=AddMembers:
    post:
        x-routing-condition-typescript: req.body.action == 'AddMembers'
        ...
/api/teams/{teamId}#action=DeleteMembers:
    post:
        x-routing-condition-typescript: req.body.action == 'DeleteMembers'
        ...

And then you should customize the server-side codgen a little bit to make use of x-routing-condition-typescript to handle those 2 operations under the same /api/teams/{teamId} post. For example, to generate code like this.

teamOperations(req: Request) {
  if (req.body.action == 'AddMembers')  //  x-routing-condition-typescript
    return this.addMembers(req);

  if (req.body.action == 'DeleteMembers')  //  x-routing-condition-typescript
    return this.deleteMembers(req);

  return new ErrorResponse(400, 'Bad Request');
}
addMembers(req) {
...
}
deleteMembers(req) {
...
}

Basically Specification Extensions allows users to provide whatever they need for codegens to generate correct code.
You can even achieve this without Specification Extensions. You can just use fragment itself to express these routing conditions and use it in your codegens to generate the server-side code , regardless of where they are, header, body, query params, etc.

/api/teams/{teamId}#headers.operation==add-members:
    post:
        ...
/api/teams/{teamId}#headers.operation==delete-members:
    post:
        ...

In essence, fragments just provide a mechanism to provide more information to allow codegens to handle scenarios that they could not handle before.

swagger is just a document, by supporting fragments doesn't mean we want to use it as it in the HTTP protocol, but to provide a way to model subordinate resources under the same path+method.

@earth2marsh

However, it is worth noting that this expansion would add some additional burden on tooling authors who would have a bigger surface area to support.

Surely there are a lot of other ways to achieve the same suppose, but I think fragments is the one that most compatible with the current standards and existing tools. For client-side, no changes are required if they follow HTTP specification strictly. If not, it is also easy to cover, you can take this one as an example: #1635 (comment)

[handrews]

@link89 With well-designed extensions you don't need to abuse fragments like this. That's the whole point of my earlier comment: #1635 (comment)

[handrews]

@OAI/tsc unless fragments are seriously under consideration for this, could we close this issue and file a new issue describing the problem rather than a (as far as I can tell) rejected solution so that discussion can be properly focused?

[darrelmiller]

Speaking just for myself, I'm not interested in trying to solve RPC issues before we address the issue of differentiating pathitems by query string.

[handrews]

@darrelmiller is there a separate issue for that? It would be good to disentangle it from all of the fragment-related discussion.

[DavidBiesack]

related: #182 ; see that. fragments are simply one mechanism people have used to deal with this (stemming from OpenAPI 2.0)

[ioggstream]

I think this proposal - while shows a clever way to solve a real use case - may hinder interoperability and constraint the evolution of the spec.

As many of you already said, fragment is not sent to the server. This is highlighted too in httpbis-semantics

curl -v http://localhost:5000/foo#ciao
> GET /foo HTTP/1.1                              <--- no fragment here
> Host: localhost:5000

imho:

• @link89 an issue should be filed for the actual use-case it tries to solve, though.
• @darrelmiller @MikeRalphson I'm for closing :)

[ioggstream]

I just found the following text

other Specifications MUST NOT define structure within the fragment identifier,
unless they are explicitly defining one for reuse by media types in their definitions
(for example, as JSON Pointer [RFC6901] does)

in https://www.rfc-editor.org/rfc/rfc8820.html#name-uri-fragment-identifiers

[link89]

https://www.rfc-editor.org/rfc/rfc8820.html#name-uri-fragment-identifiers

I think this should end the discussion.