Fix Client doesn't recover from remote connection resets

[avishniakov]

Describe changes

I tried to fix
('Connection aborted.', RemoteDisconnected('Remote end closed connection without response') error.

This bug is hardly reproducible, so I didn't come up with any test case to cover it. I would appreciate some community testing before merging this.

Pre-requisites

Please ensure you have done the following:

• I have read the CONTRIBUTING.md document.
• If my change requires a change to docs, I have updated the documentation accordingly.
• If I have added an integration, I have updated the integrations table and the corresponding website section.
• I have added tests to cover my changes.
• I have based my new branch on develop and the open PR is targeting develop. If your branch wasn't based on develop read Contribution guide on rebasing branch to develop.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Other (add details above)

[htahir1]

One way of trying to reproduce it is to have a long running pipeline in a remote orchestrator

[avishniakov]

One way of trying to reproduce it is to have a long running pipeline in a remote orchestrator

Yeah, @AlexejPenner gave me that hint, but I was not lucky in catching this still.

[stefannica]

Are you sure this is ok ? From what I can tell based on the Adapter constructor, this field needs to be set to a urllib.util.retry.Retry object instead of an integer:

    def __init__(
        self,
        pool_connections=DEFAULT_POOLSIZE,
        pool_maxsize=DEFAULT_POOLSIZE,
        max_retries=DEFAULT_RETRIES,
        pool_block=DEFAULT_POOLBLOCK,
    ):
        if max_retries == DEFAULT_RETRIES:
            self.max_retries = Retry(0, read=False)
        else:
            self.max_retries = Retry.from_int(max_retries)

The second question is what exactly is being retried ? Does an HTTP error like 404 Not Found or 403 Forbidden also trigger the client to retry the request ? All HTTP errors are treated separately by ZenML code, and therefore this retry mechanism should only be triggered by TCP errors, such as a connection being closed unexpectedly or timing out.

[schustmi]

I implemented something similar here. The Retry object also allows you to specify in which cases it retries.

[avishniakov]

Yeah, it works also with int, but then sets it as total. I updated it to retry only on connect issues.

[github-actions]

E2E template updates in examples/e2e have been pushed.

[htahir1]

Haha dont know why im tagged but ill approve it as it looks good to me

[avishniakov]

Haha dont know why im tagged but ill approve it as it looks good to me

Cause you voluntarily commented before 🙂

[stefannica]

Alright !

[github-actions]

E2E template updates in examples/e2e have been pushed.

[github-actions]

NLP template updates in examples/nlp-case have been pushed.