Clarification: Ordering of media types with parameters

[arucard21]

I think a clarification is needed in the JAX-RS specification when it comes to dealing with media types that may include parameters. I'll refer to the 3.0 specification but, if possible, this clarification would also be useful in the 2.x specification.

From the JAX-RS 3.0 specification (chapter 3.7.2, step 2b)

We write 𝑛1/𝑚1;q=𝑣1;qs=𝑣′1;d=𝑣″1≥𝑛2/𝑚2;q=𝑣2;qs=𝑣′2;d=𝑣″2 if one of these ordered conditions holds:
i. 𝑛1/𝑚1≻𝑛2/𝑚2 where the partial order ≻ is defined as 𝑛/𝑚≻𝑛/*≻*/*,
[...]

From the JAX-RS 3.0 specification (chapter 3.8, step 7)

Sort 𝑀 in descending order, with a primary key of specificity (n/m>n/*>*/*), a secondary key of q-value and a tertiary key of qs-value.

Both the request matching and media type determining algorithm require the media types to be sorted in descending order of specificity. This is described as n/m > n/* > */* in both algorithms. However, this does not cover the usage of parameters in media types. I think a clarification is needed to describe how exactly to sort a media type that has parameters.

My own interpretation is that a media type with parameters is more specific than a media type with parameters. So I would clarify this by describing the sorting as n/m;a=b > n/m > n/* > */*. This also matches the algorithm defined for the Accept HTTP header:

Media ranges can be overridden by more specific media ranges or specific media types. If more than one media range applies to a given type, the most specific reference has precedence. For example, Accept: text/*, text/plain, text/plain;format=flowed, */* have the following precedence:

1. text/plain;format=flowed
2. text/plain
3. text/*
4. */*

If this clarification is added, this may also need to included in the TCK to ensure consistency in the implemented algorithms.

Currently, the implementation in Jersey 2.x ignores all parameters except q and qs (and maybe charset) in these algorithms. So text/plain;format=flowed is considered equivalent to text/plain and would be sorted on the same level. I expect this to still be the same in Jersey 3.x though I have not tested that.

I noticed that #446 is somewhat related. It covers some other points as well but also highlights this problem

There is no precise definition of "more specific"

Since that issue was much older and did not cover this problem explicitly, I created this new issue instead of adding to that older one.

[spericas]

From the JAX-RS 3.0 specification (chapter 3.7.2, step 2b)

We write 𝑛1/𝑚1;q=𝑣1;qs=𝑣′1;d=𝑣″1≥𝑛2/𝑚2;q=𝑣2;qs=𝑣′2;d=𝑣″2 if one of these ordered conditions holds:
i. 𝑛1/𝑚1≻𝑛2/𝑚2 where the partial order ≻ is defined as 𝑛/𝑚≻𝑛/≻/*,
[...]

From the JAX-RS 3.0 specification (chapter 3.8, step 7)

Sort 𝑀 in descending order, with a primary key of specificity (n/m>n/>/*), a secondary key of q-value and a tertiary key of qs-value.

Just to be clear, these two sections are there for different purposes. Section 3.7.2 is about matching and Section 3.8 about determining the media type of responses (way after matching, of course).

Both the request matching and media type determining algorithm require the media types to be sorted in descending order of specificity. This is described as n/m > n/* > */* in both algorithms. However, this does not cover the usage of parameters in media types. I think a clarification is needed to describe how exactly to sort a media type that has parameters.

Section 3.7.2 only deals with q and qs at this time, as these relate to each other and have well-defined semantics. As for Section 3.8, this is about determining the content-type of the response. Can you elaborate the relevancy of parameters in this case?

My own interpretation is that a media type with parameters is more specific than a media type with parameters. So I would clarify this by describing the sorting as n/m;a=b > n/m > n/* > */*.

What use case would this new rule be applicable for?

If this clarification is added, this may also need to included in the TCK to ensure consistency in the implemented algorithms.

Currently, the implementation in Jersey 2.x ignores all parameters except q and qs (and maybe charset) in these algorithms. So text/plain;format=flowed is considered equivalent to text/plain and would be sorted on the same level. I expect this to still be the same in Jersey 3.x though I have not tested that.

So the intent here would be to have two resource methods one with and one without format=flowed? Essentially making all params be part of the matching process (perhaps with the exception of q and qs)?

[arucard21]

My intent is indeed to make the params part of the matching process as well as the response media type determination. I'll try to illustrate the need for this with some examples (written freehand so forgive any minor mistakes).

I'll continue with the text/plain;format=flowed example, though my own need for this actually comes from the profile parameter with application/hal+json (as well as the profile parameter on application/ld+json). In all fairness, I don't know much about this format parameter, it was used in the specs and seems simple enough to understand for this example.

@Path("examples")
public class Example {
  @GET
  @Produces("text/plain;format=flowed;qs=0.8")
  public String fixed(){
    // some implementation that returns fixed-format text
  }

  @GET
  @Produces("text/plain;format=fixed")
  public String flowed(){
    // some implementation that returns flowed-format text
  }
}

With this setup, if you do a GET request to "/examples" with Accept header set to text/plain;format=flowed, the matching algorithm (from 3.7.2) would match to the fixed() resource method instead of flowed(). Since the media type parameters are ignored, it only uses the qs value to match them which will always match to the fixed format method that has a heavier weight (default qs value of 1).

@Path("examples")
public class Example {
  @GET
  @Produces({"text/plain;format=flowed;qs=0.8","text/plain;format=fixed"})
  public String fixed(@Context Request request){
    // availableVariants is a list of 2 variants, one for each of the media types in Produces (bit much to write out in code)
    Variant selected = request.selectVariant(availableVariants);
  }
}

In this example, selectVariant() will have the same problem as the matching algorithm. It will select the fixed media type, even if the flowed mediatype was requested. I think that selectVariant() corresponds to the algorithm from 3.8 (determining the response media type) but correct me if I'm wrong.

I hope that these example help illustrate the need to distinguish media types based on the parameters as well. With the profile parameter for application/hal+json (and application/ld+json), you can actually get an entirely different API resource based on the value of that parameter.

I hope this answered the questions you asked. If anything is still unclear, please let me know.

[spericas]

I'll continue with the text/plain;format=flowed example, though my own need for this actually comes from the profile parameter with application/hal+json (as well as the profile parameter on application/ld+json). In all fairness, I don't know much about this format parameter, it was used in the specs and seems simple enough to understand for this example.

@Path("examples")
public class Example {
  @GET
  @Produces("text/plain;format=flowed;qs=0.8")
  public String fixed(){
    // some implementation that returns fixed-format text
  }

  @GET
  @Produces("text/plain;format=fixed")
  public String flowed(){
    // some implementation that returns flowed-format text
  }
}

With this setup, if you do a GET request to "/examples" with Accept header set to text/plain;format=flowed, the matching algorithm (from 3.7.2) would match to the fixed() resource method instead of flowed().

Correct.

Since the media type parameters are ignored, it only uses the qs value to match them which will always match to the fixed format method that has a heavier weight (default qs value of 1).

Yes, except the client can always use q to influence the matching. The total ordering on combined types in the spec will honor this.

@Path("examples")
public class Example {
  @GET
  @Produces({"text/plain;format=flowed;qs=0.8","text/plain;format=fixed"})
  public String fixed(@Context Request request){
    // availableVariants is a list of 2 variants, one for each of the media types in Produces (bit much to write out in code)
    Variant selected = request.selectVariant(availableVariants);
  }
}

In this example, selectVariant() will have the same problem as the matching algorithm. It will select the fixed media type, even if the flowed mediatype was requested. I think that selectVariant() corresponds to the algorithm from 3.8 (determining the response media type) but correct me if I'm wrong.

Yes.

I hope that these example help illustrate the need to distinguish media types based on the parameters as well. With the profile parameter for application/hal+json (and application/ld+json), you can actually get an entirely different API resource based on the value of that parameter.

I certainly understand where you're coming from. But, every addition brings additional complexity to the algorithm and, to be honest, the benefits seem very small in this case. Most scenarios can be deal with some use of q and qs. I'd be good to get input from others on your suggestion.

[arucard21]

Thanks for the clear and concise responses. It really helps with ensuring that the message came across well.

Yes, except the client can always use q to influence the matching. The total ordering on combined types in the spec will honor this.

This part is unclear to me. In my example, the client requests the text/plain;format=flowed media type which is equivalent to text/plain;format=flowed;q=1.0 since it defaults to the q-value of 1.0. And in this case the API would return the fixed format. So I don't see how I can use the q-value in the client to get the API to return the flowed format instead of fixed.

Would it work if I explicitly specify the fixed format at a lower priority in my Accept header? Something like text/plain;format=flowed;q=1.0,text/plain;format=fixed;q=0.1? I don't think I've tested this, so I'm not sure what this does. From the specs, it seems like this should work. Intuitively, it seems strange to me that I would have to add a media type to the Accept header just to indicate that I don't want it. This seems like a workaround.

As to whether the additional complexity is beneficial, I think the fact that this is included in the algorithm for the Accept header (in the HTTP spec) is an indication that it is. Especially since JAX-RS is intended to work with HTTP content negotiation (which uses the Accept header). But as you said, it's good to get input from others on this.

[spericas]

Thanks for the clear and concise responses. It really helps with ensuring that the message came across well.

Yes, except the client can always use q to influence the matching. The total ordering on combined types in the spec will honor this.

This part is unclear to me. In my example, the client requests the text/plain;format=flowed media type which is equivalent to text/plain;format=flowed;q=1.0 since it defaults to the q-value of 1.0. And in this case the API would return the fixed format. So I don't see how I can use the q-value in the client to get the API to return the flowed format instead of fixed.

Yes, you're correct, the default value of q does not help here, you'd need to make it explicitly.

Would it work if I explicitly specify the fixed format at a lower priority in my Accept header? Something like text/plain;format=flowed;q=1.0,text/plain;format=fixed;q=0.1? I don't think I've tested this, so I'm not sure what this does.

Right, this is how you'd need to do it. This should work if the implementation follows the algorithm in the spec.

From the specs, it seems like this should work. Intuitively, it seems strange to me that I would have to add a media type to the Accept header just to indicate that I don't want it. This seems like a workaround.

Yes, it's a workaround, but you're use case is far from common.

As to whether the additional complexity is beneficial, I think the fact that this is included in the algorithm for the Accept header (in the HTTP spec) is an indication that it is. Especially since JAX-RS is intended to work with HTTP content negotiation (which uses the Accept header). But as you said, it's good to get input from others on this.

As I said, I'd like to hear the opinion of others here.

[arucard21]

It doesn't seem like anyone else has been able to give their opinion yet. But I've been trying out this workaround (in Jersey), and it doesn't seem to work that well.

When I have a JAX-RS method that produces both media types, the selectVariant() method still only picks up the first.

And when you have separate JAX-RS methods for each media type, the problem gets even bigger. Since the JAX-RS matching algorithm doesn't consider the media type parameters as differentiating factors for media types, Jersey considers those separate JAX-RS methods as producing the same media type. So when the JAX-RS application is started, it fails with the following error.
[FATAL] A resource model has ambiguous (sub-)resource method for HTTP method GET and input mime-types as defined by"@Consumes" and "@Produces" annotations at Java methods <method1> and <method2> at matching regular expression <path>. These two methods produces and consumes exactly the same mime-types and therefore their invocation as a resource methods will always fail.

Since it's not possible to create separate JAX-RS methods for each media type and selectVariant() doesn't accurately return the best matching media type, I have no way of knowing which media type was matched. I can only try to go through the Accept header and the producible media types with custom code to find out which one should have been matched.

I hope that we can soon get feedback from others on this. Do people need to get notified to provide this feedback, or does it just take time?

[chkal]

I certainly understand where you're coming from. But, every addition brings additional complexity to the algorithm and, to be honest, the benefits seem very small in this case. Most scenarios can be deal with some use of q and qs. I'd be good to get input from others on your suggestion.

I agree with @spericas that the actual benefit of supporting this case is rather small. However, given the fact that the HTTP spec explicitly mention this kind of ordering, it may be worth to support it.

See: https://www.rfcreader.com/#share60ab9039ff7fe20a44dba01e_line1712

[arucard21]

@chkal Thanks for your input on this. I appreciate that you think this is worth supporting due to it being explicitly part of the HTTP spec. But I still want to explain why I think the benefit of supporting this use case may actually be bigger than you expect. Though it's more a matter of the drawback of not supporting this use case being more problematic than you might expect (I emphasized it in the text below, if you want to skip the recap which I added for clarity).

To recap, the problem here is that the JAX-RS spec ignores the media type parameters when comparing media types. This results in the JAX-RS algorithms simply using the first media type in the list, even when media types with better matching parameters are available. The workaround suggested by @spericas was to have the client add the non-preferred media types to the HTTP Accept header with a much lower q-value. Unfortunately, this doesn't seem to work. It works when matching a request to a method but not with the Request.selectVariant() method, the need for which I will explain further below. Another problem is with the implementation of the API itself. I provided this example earlier in the conversation:

@Path("examples")
public class Example {
  @GET
  @Produces("text/plain;format=flowed;qs=0.8")
  public String fixed(){
    // some implementation that returns fixed-format text
  }

  @GET
  @Produces("text/plain;format=fixed")
  public String flowed(){
    // some implementation that returns flowed-format text
  }
}

As it turns out, this implementation is actually not possible (at least with Jersey). The problem is that, since media type parameters are ignored, these 2 methods are considered to have the same value for @Produces. This makes it ambiguous for JAX-RS to match a request to these methods and the application will fail to start (at least with Jersey). So if you then go with the implementation that is possible, you would end up with something like this:

@Path("examples")
public class Example {
  @GET
  @Produces({"text/plain;format=flowed;qs=0.8","text/plain;format=fixed"})
  public String fixed(@Context Request request){
    // availableVariants is a list of 2 variants, one for each of the media types in Produces (bit much to write out in code)
    Variant selected = request.selectVariant(availableVariants);
  }
}

With this, the application starts up correctly but you have to determine inside the method which media type was actually resolved in order to return the correct media type. For this, you have the Request Context object with the selectVariant() method. Unfortunately, this also adheres to JAX-RS spec and ignores the media type parameter so it will not return the best matching media type.

The only way I have found to get the best matching media type, taking into account its media type parameters, is to write my own code for this in the API. And the problem with this is that there is no way, that I have found at least, to get the producible media types for a request from the JAX-RS application. So you also have to use reflection to scan for the @Produces annotations. I think we can agree that, while this works, this is a fragile approach at best and should not be considered a good practice.

This brings me to the big drawback of not supporting this use case. With the situation being as I explained above, it essentially makes media types like application/hal+json and application/ld+json unusable with JAX-RS. This is especially unfortunate given that these media types are specifically designed for use in REST APIs, which is exactly what JAX-RS is for.

These media types use the profile and profiles media type parameter respectively, to differentiate between different types of resources. Different to the extent that they may have nothing to do with each other. So when you ignore those parameters, you make it impossible to differentiate between these entirely different resources.

I hope that supporting such media types that are intended REST APIs, just like JAX-RS, is considered a sufficient benefit to support this use case.

[arucard21]

I recently noticed that this same problem was reported for Spring MVC. I'm mentioning it because I think it shows that this is a fairly common use case that is currently not being supported (by either JAX-RS or Spring).

This issue can also be stated as an inconsistency in the spec itself. Appendix B states that the HTTP Accept and Content-type headers are supported through the algorithms mentioned before. But this is currently only partially supported, since the HTTP specification includes media types with parameters while JAX-RS doesn't. So if this mismatch between the JAX-RS spec and HTTP spec is not intended to be clarified or resolved, it should at least be mentioned explicitly with good reason as to why this mismatch is allowed/needed/unavoidable.

[arucard21]

Is there any update on this?

I noticed that the Spring issue has been closed without any changes being implemented for it. But the issue itself does contain a lot more real-world use cases and links many other issues that are created, e.g. for application servers that provide implementations for both JAX-RS and Spring. This shows that this is indeed a problem that users are having.