Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor: Move synchronisation logic to LifecycleManger #1005

Merged
merged 5 commits into from
Aug 14, 2024

Conversation

morgsmccauley
Copy link
Collaborator

@morgsmccauley morgsmccauley commented Aug 12, 2024

This PR moves the synchronisation logic within ExecutorsHandler/BlockStreamsHandler in to LifecycleManager. The motive behind this is to provide greater awareness/control over the individual sync tasks (managing unhealthy, reconfiguring, etc.) to LifecycleManger.

I plan to build on top of this in my next PR, and have LifecycleManager count how many times an unhealthy stream/executor was restarted, and move the Indexer to Suspended if it goes above a configured threshold. This refactor makes "counting" actually possible.

Wanted to land the refactor PR first to make review easier.

@morgsmccauley morgsmccauley requested a review from a team as a code owner August 12, 2024 03:49
@morgsmccauley morgsmccauley linked an issue Aug 12, 2024 that may be closed by this pull request
Copy link
Collaborator

@darunrs darunrs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we retain the restart timeout until the retry configuration is in place? The timeout of retries originally helped in two cases: Block streams which would restart over incorrect stale checks, and executors which constantly crash. The former is partially remediated due to the stale check being longer. But the latter is still an issue. Basically, I believe these indexers make some async call which fails, and it escapes our try catches for some reason. As a result, it crashes. This dumps logs into Runner and is also unnecessarily taxing on the system as restarts involve block prefetches. It also messes with the metrics. I have not gotten around to fixing any of those issues yet. Keeping in the timeout on retries at least mitigates the issue until we either have retry policies in place.

coordinator/src/handlers/block_streams.rs Outdated Show resolved Hide resolved
coordinator/src/handlers/executors.rs Outdated Show resolved Hide resolved
coordinator/src/handlers/executors.rs Outdated Show resolved Hide resolved
@morgsmccauley
Copy link
Collaborator Author

Ah sorry, I meant to add the timeout back in but forgot - will do that now!

@morgsmccauley morgsmccauley merged commit 4b46f54 into main Aug 14, 2024
4 checks passed
@morgsmccauley morgsmccauley deleted the refactor/move-sync-logic-to-lifecycle branch August 14, 2024 22:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Suspend stalled indexers after X restarts
2 participants