DAOS-16276 doc: Address engine unavailability #15456
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Ticket title is 'Document how to bring DAOS cluster online after many servers failed' |
Add a section on handling unavailable engines. Signed-off-by: Li Wei <wei.g.li@intel.com> Required-githooks: true
|
Sorry, a minor change: Added "that remain" at the end of a paragraph. |
kccain
reviewed
Nov 6, 2024
docs/admin/troubleshooting.md
Outdated
| @@ -554,6 +554,21 @@ To resolve the issue: | |||
|
|
|||
| Alternately, the administrator may erase and re-format the DAOS system to start over fresh using the new addresses. | |||
|
|
|||
| ### Engines become unavailable | |||
|
|
|||
| Engines may become unavailable due to server power losses and reboots, network switch failures, etc. After staying unavailable for a certain period of time, these engines may become "excluded" or "errored" in `dmg system query` output. Once the states of all engines stablize (see [`CRT_EVENT_DELAY`](env_variables.md)), each pool will check whether there is enough redundancy (see [Pool RF](pool_operations.md#pool-redundancy-factor)) to tolerate the unavailability of the "excluded" or "errored" engines. If there is enough redundancy, these engines will be excluded from the pool ("disabled ranks" in `dmg pool query --health-only` output); otherwise, the pool will perform no exclusion and may become temporarily unavailable. Similarly, when engines become available, whenever the states of all engines stablize, each pool will perform the aforementioned check for any unavailable engines that remain. | |||
There was a problem hiding this comment.
suggest adding to the end of the sentence
... "and may become temporarily unavailable (as seen by timeouts of certain commands, for example, dmg pool query and dmg pool list)
There was a problem hiding this comment.
I asked myself this too when writing these words. Added.
There was a problem hiding this comment.
Also fixed "stablize" -> "stabilize".
Signed-off-by: Li Wei <wei.g.li@intel.com> Required-githooks: true
liuxuezhao
approved these changes
Nov 7, 2024
kccain
approved these changes
Nov 7, 2024
liw
added a commit
that referenced
this pull request
Nov 13, 2024
Add a section on handling unavailable engines. Signed-off-by: Li Wei <wei.g.li@intel.com> Required-githooks: true
daltonbohning
pushed a commit
that referenced
this pull request
Dec 12, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add a section on handling unavailable engines.
Before requesting gatekeeper:
Features:(orTest-tag*) commit pragma was used or there is a reason documented that there are no appropriate tags for this PR.Gatekeeper: