Behaviour improvements when the ipfs daemon is unavailable #1762
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit introduces unlimited waiting on start until a request to `ipfs id` succeeds. Waiting has some consequences: * State watching (recover/sync) and metrics publishing does not start until ipfs is ready * swarm/connect is not triggered until ipfs is ready. Once the first request to ipfs succeeds everything goes to what it was before. This alleviates trying operations like sending our IDs in metrics when IPFS is simply not there.
Unfortunately we were not paying attentions to errors while rpc-streaming pins in the pintracker. The result is that the StatusAll operation would list all the pins as unexpectedly unpinned when ipfs is offline, and this would result in recover/requeing operations for all pins when ipfs is offline. This commits changes the behaviour so that if IPFS Pin/ls has resulted in an error, then the StatusAll operation cannot complete at all.
This fixes a bug in API code that made it return 204-No content when the RPC methods failed with an error before any items were returned on the channel.
When IPFS starts failing or doesn't respond (i.e. during a restart), cluster is likely to start sending requests at very fast rates. i.e. if there are 100k items to be pinned, and pins start failing immediately, cluster will consume the pin queue really fast and it will all be failures. At the same time, ipfs is hammered non-stop until recover, which may make it harder. This commits introduces a rate-limit when requests to IPFS fail. After 10 failed requests, requests will be sent at most at 1req/s rate. Once a requests succeeds, the rate-limit is raised. This should prevent hammering the IPFS daemon, but also increased CPU in cluster as it burns through pinning queues when IPFS is offline, making the situation in machines worse (and emitting way more logs).
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR introduces a series of improvements in how the cluster peer deals with non-responding IPFS daemons:
This should improve the cluster peer behaviour during IPFS issues, which right now results in endless retrying and associated CPU/memory increases, which do not help IPFS recover or restart swiftly.
Fixes #1733.