Pushing to registry is timing out

[alexzimmer96]

Description of the issue:
Our docker-registry is sometimes a little bit slow, so the building-process inside our CI-pipeline fails with the message Build image failed: Read timed out. The timeout i selected inside my settings.xml for this server is not respected.

Expected behavior:
Respecting the Timeout inside the servers-tag in settings.xml

Steps to reproduce:
Try to push an image to an slow docker-registry. Then set the timeout for this registry inside your settings.xml and test ist again. You will see that the timeout is not respected.

Tested with version 0.9.6

[coollog]

@alexanderzimmer Thanks for reporting this issue! We'll look into using the timeout from the settings.xml. Is your timeout configuration in the old or new wagon format? (https://maven.apache.org/guides/mini/guide-http-settings.html#Connection_Timeouts)

[alexzimmer96]

@coollog Thanks for your reply! I've tried both and ended up using:

            <configuration>
                <httpConfiguration>
                    <all>
                        <connectionTimeout>60000</connectionTimeout>
                        <readTimeout>60000</readTimeout>
                    </all>
                </httpConfiguration>
            </configuration>

But it still does not work.

[coollog]

@alexanderzimmer Yea it will not work in the current state because we do not read timeout configuration from Maven settings. We might support the format in your comment.

@GoogleContainerTools/java-tools Any ideas on other ways of solving this issue?

[patflynn]

I'm not convinced that the maven http setting configuration should be applied to the container registry interactions. Perhaps we should provide our own configuration for this?

…

On Thu, Jul 12, 2018 at 10:56 AM Q Chen ***@***.***> wrote: @alexanderzimmer <https://github.com/alexanderzimmer> Yea it will not work in the current state because we do not read timeout configuration from Maven settings. We might support the format in your comment. @GoogleContainerTools/java-tools <https://github.com/orgs/GoogleContainerTools/teams/java-tools> Any ideas on other ways of solving this issue? — You are receiving this because you are on a team that was mentioned. Reply to this email directly, view it on GitHub <#582 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHf5HSAozSNJMqV5Pp0CMZkvhQQ3iKPTks5uF2OxgaJpZM4VMfvO> .

[chanseokoh]

I think it makes sense, because we read the <servers> info from settings.xml and get the container registry info from there (including their username and password), and it looks like the HTTP (tmieout) configuration is under <servers>.

[coollog]

We do have to think about what to do on the Gradle side too to keep the experience consistent. Another option is just to have a much higher timeout (like 60s) globally without configuration.

[patflynn]

@coollog what's it set to now?

So the container registry urls are specified in in settings.xml? Are we using an http wagon implementation for our http client?

[coollog]

Since we use the Google HTTP client, the default timeout is 20 seconds.

[chanseokoh]

So the container registry urls are specified in in settings.xml?

Not really, but in some sense yes. AFAIK (correct me if I'm wrong), we read <servers> in settings.xml only for retrieving credential info: https://github.com/GoogleContainerTools/jib/tree/v0.9.6-maven/jib-maven-plugin#using-maven-settings

[alexzimmer96]

Not really, but in some sense yes. AFAIK, (correct me if I'm wrong), we read <servers> in settings.xml only for retrieving credential info

@chanseokoh
You're right. Only credentials are readed from <servers>

[coollog]

I think we should probably provide a system property to override the default timeout for now, and consider adding configuration/maven settings support later. So with Maven you would have something like mvn jib:build -Djib.httpTimeout=60 and in Gradle you would have gradle jib -Djib.httpTimeout=60. @GoogleContainerTools/java-tools WDYT?

[chanseokoh]

I guess that will work. Just wondering if there are some guides/policies for reading various configuration/settings from where, like through -D, plugin configurations in pom.xml or build.gradle, Maven settings (including ~/.m2/settings.xml), etc. For example, in #599 (comment), we talk about sendCredentialsOverHttp as a system property, whereas we have allowinsecureregistries as plugin configuration.

[coollog]

@chanseokoh I'm not aware of any community-standard guides/policies regarding this, but I think that we should generally avoid adding additional configuration to build files for configuration that is not closely related to image builds (and especially for edge case uses). We can reconsider allowInsecureRegistries as possibly not a configuration, but it was added because it supports a large use case (local private registries without TLS).

[briandealwis]

My 2¢ is that we should support the native frontend approach as much as possible. But for projects to require contributors to apply tweaks to their ~/.m2/settings.xml to set timeouts for known problematic registries would be annoying. So perhaps we could also support having properties like jib.<registry>.httpTimeout to override settings (e.g., jib.registry.hub.docker.com.httpTimeout=9000), but the ~/.m2/settings.xml win if present?

[coollog]

@briandealwis Do you think having jib.<registry>.httpTimeout would be beneficial over just jib.httpTimeout? I was thinking that it wouldn't be harmful to have a common timeout for base and target registries, and the parameter would be easier to use. Also, we should probably call it connectionTimeout to make it explicit that this is only for connection timeout.

[briandealwis]

You're right @coollog, setting a large timeout shouldn't have any effect for fast registries.

[chanseokoh]

Also, we should probably call it connectionTimeout to make it explicit that this is only for connection timeout.

Is there a reason to target only the connection timeout and not care about the read timeout? I think in the OP's situation, timeout can happen in both connection and read, and the log says it's the read time out.

Build image failed: Read timed out

[coollog]

@chanseokoh Ah right, we should just have it as jib.httpTimeout then, and set both connection timeout and read timeout as that.

[chanseokoh]

The jib.httpTimeout system property will be available once 0.9.7 is released.

[coollog]

Hi, @alexanderzimmer we've just release version 0.9.7 - can you try with that version and using the jib.httpTimeout system property?

For example:

mvn compile jib:build -Djib.httpTimeout=60000

[alexzimmer96]

@coollog Hey, thanks for your support. This helped me a lot and is running now!

[whirlwin]

I encountered this on OpenShift, which has a concept of ImageStream which is a representation of a container image, roughly speaking.

Deleting the ImageStream solved the issue for me. ✔️

oc delete imagestream <name> -n <namespace>

[luger1990]

            <httpConfiguration>
                <all>
                    <connectionTimeout>60000</connectionTimeout>
                    <readTimeout>60000</readTimeout>
                </all>
            </httpConfiguration>
        </configuration>

this works for me