Standardized problem types don't include all allowed Status Codes

[pvdbosch]

In GitLab by @smals-chlo on Nov 12, 2019, 16:21

In the Rest Guidelines, standardized problem types are defined for Status Codes 400, 401, 403, 404 & 429.
https://www.gcloud.belgium.be/rest/#standardized-problem-types

But some other allowed Status Codes could also be standardized: 405, 406, 412 & 415 as well as the 5XX error codes.

[pvdbosch]

In GitLab by @pvdbosch on Nov 13, 2019, 17:35

Hi Chrysoula,

The idea is to use Problem JSON only when you can give additional information than provided by the HTTP status code.

One could argue that, in general, the status codes you mentioned are explanatory enough. There may be specific use cases where additional information could be useful, which could be modeled as API-specific problem types. If there are some generic use cases you can think of that require additional info, we could yet consider to add them.

A client should also not expect to always get a Problem JSON on every status code because reverse proxies or other middleboxes may generate HTTP errors outside control of the server application.

@wsalembi, could you confirm that this was the reasoning followed in the working group? If so, I can add this explicitly to the guidelines. If not, we can bring this up again.

Btw, optimistic locking for update/create scenarios with If(-None)-Match headers and status code 412 isn't well documented currently in our guidelines. Instead 409 is listed for updates with a failed optimistic locking. I'm creating another issue to followup on this.

[pvdbosch]

In GitLab by @wsalembi on Nov 14, 2019, 09:17

@pvdbosch My intention was to always return problem responses for any 4xx/5xx error status. Even if they just contain default fields, which brings a bit of consistency in any error returned by the API. The client applications can just inspect the problem response.

We have a JAX-RS default exception mapper in the backends for that purpose and on gateway we have default problem definitions too.

So +1 to standardize the missing status codes above.

[pvdbosch]

In GitLab by @pvdbosch on Nov 14, 2019, 11:58

An API Gateway or an Apache RP may also generate some of these error responses without a JSON Problem body, before arriving in the the application code.
So I'd think the client would need to be able to treat status codes without problem response anyway, and we can at most recommend API providers to try to return JSON Problems when possible.

I'll add this issue to the agenda for next work group meeting.

[pvdbosch]

I agree that Problem types could be standardized for:

• 412 Precondition Failed - for use case Optimistic Locking optimistic locking with ETags #37 - https://www.gcloud.belgium.be/rest/problems/preconditionFailed
• 500 Internal Server Error - because it can be useful to provide additional details in the Problem JSON body - https://www.gcloud.belgium.be/rest/problems/internalServerError

For 405 Method Not Allowed, 406 Not Acceptable, 415 Unsupported Media Type, I don't see much added value in standardizing the Problem types because they are self-explanatory and are most likely to be caused by programming errors and go unhandled by clients.
On the gCloud APIGW, we already had some errors (e.g. required query param missing) that couldn't be intercepted in order to create a Problem JSON response. I suspect this to be case for some of these codes as well, but haven't tested it yet.
I'm not really against standardized types for these, but I think we can only go as far as specifying that these "MAY" be supported by an API.

In JAX-RS, an exception mapper could still create a Java Problem object even when the http response didn't have a body.

[pvdbosch]

At CBSS, we're using 502 - https://api.ksz-bcss.fgov.be/rest/problems/badGateway
for problems when a CBSS API is a proxy to an external 3rd party API and that API is down. Maybe candidate to standardize as well?

[pvdbosch]

WG okayed adding problem types for 500 and 502.
I'll create a PR for this. @smals-chlo will add 502 to the overview tables as well.

Other types in use by Smals that are not in REST guide:

• 413 "payloadTooLarge" : new ProblemTemplate(BASE + "payloadTooLarge", 413, "Payload Too Large", "The request size is too big"),
• 429 "tooManyFailedRequests" : new ProblemTemplate(BASE + "tooManyRequests", 429, "Too Many Failed Requests", "No more requests accepted after previous failures."),

Both seem OK to me to standardize in REST guide. Also validate with other members of WG.
@wsalembi : tooManyFailedRequests seems to have same type URI as tooManyRequests (BASE + "tooManyRequests"); was this done on purpose?

For other http codes 405,406,415 no WG participant had a specific use case to standardize them as problem types.

For "409 Conflict", the use case is probably be dependent on business context (e.g. delete of a case refused because case is still open, action on a deceased person refused, ...). Better to define more specific problem types according to business context. Defining a generic type would discourage this.

[smals-chlo]

In fact, in the Rest Guide there is a standardized problem type for 429 Too Many Requests:
https://www.gcloud.belgium.be/rest/#too-many-requests.
I am a bit confused about tooManyFailedRequests --> Do they need to be failed for a 429 to happen? Or just Too Many (failed or not) :
The 429 status code indicates that the user has sent too many requests in a given amount of time.

[pvdbosch]

The use case is probably that the server enforces quota per user on number of requests, and an additional lower quota on number of failed requests.
When a user sends a lot of requests that are refused it may indicate a DOS attack or hacking attempt.
In both cases, Smals uses 429 but with the Problem JSON payload to indicate the difference.
But it would make sense then to use a different problem type than using the same one.

[pvdbosch]

created PR #66 to add the four new problem types

[pvdbosch]

I'll add problem types to guide:

429 gcloud.../problems/tooManyRequests
gcloud.../problems/tooManyFailedRequests
413 payloadTooLarge

[pvdbosch]

PR #66 is ready for review

[pvdbosch]

Four problem types have been added in PR #66 ; closing this issue.
A problem type for status 412 can be introduced as part of #37.