added worker error on timeout if restart fails #56
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Abuelodelanada
approved these changes
Aug 16, 2024
PietroPasotti
added a commit
that referenced
this pull request
Aug 26, 2024
* Catch SecretNotFoundError when privkey shouldn't be here (#48) * Catch SecretNotFoundError when privkey shouldn't be here * Bring back if self.tls_available check from pre-refactor times * lint * Bump the version that was forgotten in #48 (#50) * Add issues integration action (#52) * Add issues integration action * Order imports differently as lint started to complain * Update databag model dump return value (#53) * dump returns databag * tests * root ca cert patch * fixed static checks * fix * removed conftest * added health check logic to worker (#55) * added health check logic to worker * adapted status check to be less tempo-specific * use regular paths for codespell too * rerun black * vbump * added worker error on timeout if restart fails (#56) * added worker error on timeout if restart fails * maintenance status throughout retries * fixed static * only restart own services * added tls support for worker checks (#59) * added tls support for worker checks * lint * static fix * Test coordinator (#60) * Added the following: 1) Coordinator pytest fixtures 2) Coordinator unit tests 3) Rrefactoring of roles_config in coordinator.py * ClusterRolesConfig was switched to a dataclass with __post_init__ and is_coherent_with methods 4) Created test_roles_config.py tests * Chore: Fix leftover comments and minor code changes * * Updates 1) Fmt 2) Merged 'main:coordinator.py' into test-coordinator-cleanup to fix Secrets error * Added docstrings to ClusterRoleConfig --------- Co-authored-by: Mateusz Kulewicz <mateusz.kulewicz@canonical.com> Co-authored-by: PietroPasotti <starfire.daemon@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #51
Issue
The worker attempts to start the services in the layer as soon as it has all the necessary pieces of config.
If that fails, for whatever reason, it currently reports 'active' and that's it.
Any follow-up event, unless it signals config changes, will not even attempt to restart the services.
Solution
The worker will retry for 15 minutes (default, configurable) to start the services and, if it fails, raise an exception and let the charm go into error state and write some logs that tell the user what's wrong and that it's probably an issue with some external services that the workload depends upon to start, such as s3.
The idea is that juju will retry the hook and eventually as those external services come up, the issue will resolve itself.
Rejected alternatives:
resurrector something similar to wake yourself up with custom events periodically until the check finally passesContext
https://discourse.charmhub.io/t/its-probably-ok-for-a-unit-to-go-into-error-state/13022/30
Testing Instructions