ConsulCache throws java.net.SocketTimeoutException: timeout

[zhuangmg]

Hi,

When I use ServiceHealthCache to register a watch service changes, once I start the service heath cache, I got the timeout error every 10s.

    ServiceHealthCache svcHealthCache = ServiceHealthCache.newCache(healthClient, serviceName);

    ConsulCache.Listener<ServiceHealthKey, ServiceHealth> listener = new ConsulCache.Listener<ServiceHealthKey, ServiceHealth>() {
        @Override
        public void notify(Map<ServiceHealthKey, ServiceHealth> newValues) {
            //notifyListener.notify(serviceName, svcs);
        }
    };

    svcHealthCache.addListener(listener);

    svcHealthCache.start();

[OkHttp http://localhost:8500/v1/health/service/TEST001?index=793&wait=10s&passing=true] ERROR [com.orbitz.consul.cache.ConsulCache] - Error getting response from consul. will retry in 10 SECONDS
java.net.SocketTimeoutException: timeout
at com.orbitz.okio.Okio$3.newTimeoutException(Okio.java:207)
at com.orbitz.okio.AsyncTimeout.exit(AsyncTimeout.java:261)
at com.orbitz.okio.AsyncTimeout$2.read(AsyncTimeout.java:215)
at com.orbitz.okio.RealBufferedSource.indexOf(RealBufferedSource.java:306)
at com.orbitz.okio.RealBufferedSource.indexOf(RealBufferedSource.java:300)
at com.orbitz.okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:196)
at com.orbitz.okhttp3.internal.http.Http1xStream.readResponse(Http1xStream.java:184)
at com.orbitz.okhttp3.internal.http.Http1xStream.readResponseHeaders(Http1xStream.java:125)
at com.orbitz.okhttp3.internal.http.HttpEngine.readNetworkResponse(HttpEngine.java:723)
at com.orbitz.okhttp3.internal.http.HttpEngine.access$200(HttpEngine.java:81)
at com.orbitz.okhttp3.internal.http.HttpEngine$NetworkInterceptorChain.proceed(HttpEngine.java:708)
at com.orbitz.okhttp3.internal.http.HttpEngine.readResponse(HttpEngine.java:563)
at com.orbitz.okhttp3.RealCall.getResponse(RealCall.java:241)
at com.orbitz.okhttp3.RealCall$ApplicationInterceptorChain.proceed(RealCall.java:198)
at com.orbitz.okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:160)
at com.orbitz.okhttp3.RealCall.access$100(RealCall.java:30)
at com.orbitz.okhttp3.RealCall$AsyncCall.execute(RealCall.java:127)
at com.orbitz.okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at com.orbitz.okio.Okio$2.read(Okio.java:139)
at com.orbitz.okio.AsyncTimeout$2.read(AsyncTimeout.java:211)
... 18 more

[DimaGolomozy]

You are using "default" newCache method that with
watchSeconds = 10
then the default connection timeout (i think from orbitz.okhttp3) is smaller than the "watchSeconds"
this why, your connection exits with TimeoutException.
(for what i understand, watchSeconds=10, tell the consul server to send you a respond after 10 sec)

You can use 'org.glassfish.jersey.core:jersey-client:2.22.1'
that uses timeout longer than 10sec as default.

Or Build consul instance with custom connection.

Hope I'm right. and it helped you

[zhuangmg]

Thanks, use withReadTimeoutMillis(20000) to build consul client can fix the problem.

[derjust]

Also stepped on this one.
Maybe that should be added to the README.md...

[Hronom]

Me to, what the status of this? I think default values should be fixed to avoid such error...
@derjust thanks your workaround has fixed the problem, please add this as default value.

[shays10]

Yup, stepped on this one as well. Thanks for helping out!

[alugowski]

Here's why increasing the OkHttp read timeout helps.

There are two timers in a cache. One is the read timeout, which is how long OkHttp will wait for a response (OkHttp default is 10s). The second is the blocking request wait time. The client makes a call, and if there is no change then Consul waits at most this amount of time to reply "no change". consul-client default is also 10s.

But see this in their docs (https://www.consul.io/api/index.html#blocking-queries):

A small random amount of additional wait time is added to the supplied maximum wait time to spread out the wake up time of any concurrent requests. This adds up to wait / 16 additional time to the maximum duration.

What's happening is that that jitter makes Consul reply in slightly longer than 10s, which makes OkHttp time out. I think the client should use maybe 9s wait time by default to avoid this.

Note that the will retry in 10 SECONDS is yet another timeout, but irrelevant.