Feature: Provide RequestBody.create WITHOUT default charset=utf-8

[SoftwareArtisan]

• Feature Request. Start by telling us what problem you’re trying to solve. Often a solution
  already exists! Don’t send pull requests to implement new features without first getting our
  support. Sometimes we leave features out on purpose to keep the project small.

There are cases where adding charset=utf-8 breaks server-side implementations, most notably, Amazon AWS Gateway API JSON model mapping.

JSON's default encoding is UTF-8 per https://tools.ietf.org/html/rfc7158#section-8.1 so sending an "application/json" should be equivalent to "application/json; charset=utf-8".

Proposal:

Introduce another RequestBody.create OR modify MediaType to handle contentType="".

And please don't break current work-around:

public static RequestBody create(final MediaType contentType, final byte[] content

Line 46: RequestBody.java

public static RequestBody create(MediaType contentType, String content) {
    Charset charset = Util.UTF_8;
    if (contentType != null) {
      charset = contentType.charset();
      if (charset == null) {
        charset = Util.UTF_8;
        contentType = MediaType.parse(contentType + "; charset=utf-8");
      }
    }
    byte[] bytes = content.getBytes(charset);
    return create(contentType, bytes);
}

[SoftwareArtisan]

There are cases where adding charset=utf-8 breaks server-side implementations, most notably, Amazon AWS Gateway API JSON model mapping.

JSON's default encoding is UTF-8 per https://tools.ietf.org/html/rfc7158#section-8.1 so sending an "application/json" should be equivalent to "application/json; charset=utf-8".

Proposal:

Introduce another RequestBody.create OR modify MediaType to handle contentType="".

And please don't break current work-around:

public static RequestBody create(final MediaType contentType, final byte[] content

Line 46: RequestBody.java

public static RequestBody create(MediaType contentType, String content) {
    Charset charset = Util.UTF_8;
    if (contentType != null) {
      charset = contentType.charset();
      if (charset == null) {
        charset = Util.UTF_8;
        contentType = MediaType.parse(contentType + "; charset=utf-8");
      }
    }
    byte[] bytes = content.getBytes(charset);
    return create(contentType, bytes);
}

[swankjesse]

You should also ask Amazon to fix their API. Not accepting the UTF-8 charset is certainly an easy-to-fix mistake.

[SoftwareArtisan]

1. Adding the charset op to the header is not required under the spec
2. Adding it limits the API's flexibility
3. More general with specialization is the better design path
4. Get AWS to change their API? Easier to lead a camel...

[swankjesse]

You can opt-out of this behavior with the create() overload that takes a byte[]. And you should at least tell the AWS people about the problem with their API.

[SoftwareArtisan]

okHttp code would be better served by using principle of "least surprise".

In this case, a developer sets the MediaType to NADA, feeds it to RequestBody. The "least surprising" thing is the header is set as per the developers instruction not shove a charset=XXX modifier on the header - something that is NOT required by the spec.

In any case, I'll let the issue go.

[laurent22]

This issue is also causing problems when trying to access the Dropbox API from React-Native. There are also a few other examples mentioned in this thread: facebook/react-native#8237

Although the "charset=utf-8" part is technically valid, it seems to be unexpected by many servers and so breaks many APIs. In my opinion, the lib shouldn't modify the content-type in any way. If the developer sets a charset, then append it, but otherwise leave it as it is.

[matt-42]

I got exactly the same problem using the dropbox upload API from react native.

[laurent22]

@matt-42, I've contacted Dropbox support and they told me they have now raised the issue with the engineering team to see if they can change that behavior, so hopefully it will be fixed at some point.

[matt-42]

Me too :)
https://www.dropboxforum.com/t5/API-support/The-Dropbox-API-V2-not-compatible-with-React-Native/m-p/226210#M12338

[ceefour]

This should work:

Content-Type: image/jpeg
Content-Encoding: base64

With okhttp it becomes Content-Type: image/jpeg; charset=utf-8, which breaks stuff (e.g. AWS S3 presigned URL).

I agree with @SoftwareArtisan . And regarding using byte[] as @swankjesse suggested: Base64 is a string, it's even ASCII.

If okhttp wishes to keep its current behavior, would at least it be possible to honor MIME type parameters and Content-Encoding? For example, application/json does not accept charset parameter and neither does image/jpeg (or any image/*).

[yschimke]

The workaround @swankjesse suggested is still working. As such, personally I think OkHttp should continue to do the most correct thing in the default cases.

    val client = OkHttpClient()
    val body = RequestBody.create(MediaType.parse("application/json"), "{}".toByteArray(Charsets.US_ASCII))
    val request = Request.Builder().url("https://httpbin.org/post").post(body).build()
    var response = client.newCall(request).execute()

    println(response.body()!!.string())

{
  "args": {}, 
  "data": "{}", 
  "files": {}, 
  "form": {}, 
  "headers": {
    "Accept-Encoding": "gzip", 
    "Connection": "close", 
    "Content-Length": "2", 
    "Content-Type": "application/json", 
    "Host": "httpbin.org", 
    "User-Agent": "okhttp/3.9.1"
  }, 
  "json": {}, 
  "origin": "82.5.95.16", 
  "url": "https://httpbin.org/post"
}

@ceefour is "Content-Encoding: base64" valid and common in some APIs?

[ceefour]

@yschimke the problem is not everybody is able to access okhttp directly, e.g. people coming from React Native, especially CRNA and Expo where accessing Java code is not possible.

There are a lot of people using RN/CRNA/Expo. Base64 is currently ubiquitous in the RN world due to weird RN Networking APIs (which lead to facebook/react-native#8237); and AWS S3 supports it.

However, my point is not about Base64 specifically. It's that there are only limited Content-Types where charset= parameter makes sense (this is not me saying, but the specs), and that okhttp needs to conform to them. The specs say that application/json, image/*, application/octet-*, binary/*, so okhttp shouldn't make it up for those content types (and breaking apps in the process); regardless of whether a string or byte[] is passed.

I'd argue that setting Content-Type properly is developer's job unless he explicitly wants okhttp to decide, but I understand if my argument is unacceptable. I also understand that for content types like text/plain, text/html, etc. having the Content-Type charset added automatically shouldn't be a problem. But for the types I mentioned above, mangling it does break things.

[yschimke]

image/* won't have it set unless you also base64 encode and send as a string, at which point arguably you should specify ASCII anyway?

If you send images as byte[] it won't happen. So it seems like this is where the handling should happen, not in OkHttp.

[swankjesse]

I think the right fix here is to change React Native to offer an API that does what you need. It's unfortunate that your frameworks don't give you access to the features that do what you want; changing OkHttp to accommodate this is intractible.

[ceefour]

@swankjesse I agree that it's possible for React Native to use byte[]'s API, however it doesn't mean that current okhttp approach is correct for all circumstances.

Consider the original request, application/json does not accept charset parameter - RFC 7158 section 11. Regardless whether the content body is a string or byte[], a client should not append a charset parameter for the simple reason that the MIME type doesn't support it.

[ceefour]

@yschimke "image/* won't have it set unless you also base64 encode and send as a string, at which point arguably you should specify ASCII anyway?"

No, image/*; charset=ASCII is not correct. ASCII is the charset of the content encoding (Base64) not the content type. For example, it's possible (however weird):

Content-Type: text/html; charset=utf-8
Content-Encoding: base64

To get the HTML string, what you'd do is:

1. decode the base64-encoded content, then you get bytes
2. decode the bytes as UTF-8

[yschimke]

@ceefour That is fine and supported. You should just provide a byte[] instead of asking OkHttp to do the string encoding, and add the charset parameter for you. OkHttp doesn't currently encode specific logic for all these MediaTypes. i.e. rfc7158 isn't considered by OkHttp currently, but the library doesn't contradict if if used properly.

[Jitsusama]

I have come across at least 2 vendor APIs that have similar issues. They are proprietary APIs. When I request that they change their API to accommodate common standards, they say no. While it might not be ideal, it is a common issue when dealing with eclectic HTTP APIs that specifying a charset will cause the request to fail.

[programadornatal]

Use axios to resolve this:
https://github.com/qiangmao/axios

[PhilippHeuer]

I just spend a lot of time on this because i didn't realize okhttp adds headers i don't know about ...
A mitm proxy finally showed me what happend so i found this.

Please add a note in the method description, so that we know whats going on inside.

[sharonbn1]

Having the same issue with MS Dynamics API.

[righel]

Had the same problem with Amazon API Gateway, my workaround was using addNetworkInterceptor to overwrite the Content-Type header:

FixContentTypeInterceptor.java:

import okhttp3.*;
import java.io.IOException;

public final class FixContentTypeInterceptor implements Interceptor {
    @Override public Response intercept(Interceptor.Chain chain) throws IOException {
        Request originalRequest = chain.request();

        Request fixedRequest = originalRequest.newBuilder()
                .header("Content-Type", "application/json")
                .build();
        return chain.proceed(fixedRequest);
    }

}

Main.java:

[...]
MediaType JSON = MediaType.get("application/json; charset=utf-8");
String json = "{}";
OkHttpClient client = new OkHttpClient.Builder()
                .addNetworkInterceptor(new FixContentTypeInterceptor())
                .build();
RequestBody body = RequestBody.create(json, JSON);
Request request = new Request.Builder()
                .url(url)
                .post(body)
                .build();
[...]