Pods upgrade: log a summary of failed predicates #2128
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
thbkrkr
reviewed
Nov 18, 2019
| "Nodes can't be upgraded", | ||
| "namespace", ctx.es.Namespace, | ||
| "es_name", ctx.es.Name, | ||
| "failedPredicates", groupByPredicates(failedPredicates)) |
There was a problem hiding this comment.
Nit: maybe stay in snake case to be consistent with the other fields: failed_predicates.
pebrc
reviewed
Nov 18, 2019
| // the user to understand why. | ||
| if len(failedPredicates) > 0 { | ||
| log.Info( | ||
| "Nodes can't be upgraded", |
There was a problem hiding this comment.
Can we rephrase this? Something like "Cannot proceed with upgrade of some nodes at this time" Something that makes it sound less final?
pebrc
reviewed
Nov 18, 2019
| predicate string | ||
| } | ||
|
|
||
| func newFailedPredicate(pod, predicate string) *failedPredicate { |
There was a problem hiding this comment.
Not sure if the non-exported constructor on a non-exported trivial struct adds much?
There was a problem hiding this comment.
Sorry for the drive-by comments!
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Follow up of #2099
This pr reduces the verbosity of the logs when nodes can't be restarted because of some failing predicates:
before:
{"level":"info","@timestamp":"2019-11-18T13:36:45.996Z","logger":"elasticsearch-controller","message":"Starting reconciliation run","ver":"1.0.0-beta1-16b253de","iteration":7,"namespace":"default","name":"elasticsearch-sample"} {"level":"info","@timestamp":"2019-11-18T13:36:46.282Z","logger":"kibana-association-controller","message":"Ending reconciliation run","ver":"1.0.0-beta1-16b253de","iteration":6,"namespace":"default","name":"kibana-sample","took":0.287476481} {"level":"info","@timestamp":"2019-11-18T13:36:46.810Z","logger":"zen2","message":"Ensuring no voting exclusions are set","ver":"1.0.0-beta1-16b253de","namespace":"default","es_name":"elasticsearch-sample"} {"level":"info","@timestamp":"2019-11-18T13:36:46.875Z","logger":"driver","message":"Predicate failed","ver":"1.0.0-beta1-16b253de","pod_name":"elasticsearch-sample-es-default-7","predicate_name":"do_not_restart_healthy_node_if_not_green"} {"level":"info","@timestamp":"2019-11-18T13:36:46.875Z","logger":"driver","message":"Predicate failed","ver":"1.0.0-beta1-16b253de","pod_name":"elasticsearch-sample-es-default-6","predicate_name":"do_not_restart_healthy_node_if_not_green"} {"level":"info","@timestamp":"2019-11-18T13:36:46.875Z","logger":"driver","message":"Predicate failed","ver":"1.0.0-beta1-16b253de","pod_name":"elasticsearch-sample-es-default-5","predicate_name":"do_not_restart_healthy_node_if_not_green"} {"level":"info","@timestamp":"2019-11-18T13:36:46.875Z","logger":"driver","message":"Predicate failed","ver":"1.0.0-beta1-16b253de","pod_name":"elasticsearch-sample-es-default-4","predicate_name":"do_not_restart_healthy_node_if_not_green"} {"level":"info","@timestamp":"2019-11-18T13:36:46.875Z","logger":"driver","message":"Predicate failed","ver":"1.0.0-beta1-16b253de","pod_name":"elasticsearch-sample-es-default-3","predicate_name":"do_not_restart_healthy_node_if_not_green"} {"level":"info","@timestamp":"2019-11-18T13:36:46.875Z","logger":"driver","message":"Predicate failed","ver":"1.0.0-beta1-16b253de","pod_name":"elasticsearch-sample-es-default-2","predicate_name":"do_not_restart_healthy_node_if_not_green"} {"level":"info","@timestamp":"2019-11-18T13:36:46.875Z","logger":"driver","message":"Predicate failed","ver":"1.0.0-beta1-16b253de","pod_name":"elasticsearch-sample-es-default-1","predicate_name":"do_not_restart_healthy_node_if_not_green"} {"level":"info","@timestamp":"2019-11-18T13:36:46.875Z","logger":"driver","message":"Predicate failed","ver":"1.0.0-beta1-16b253de","pod_name":"elasticsearch-sample-es-default-0","predicate_name":"do_not_restart_healthy_node_if_not_green"} {"level":"info","@timestamp":"2019-11-18T13:36:46.891Z","logger":"elasticsearch-controller","message":"Updating status","ver":"1.0.0-beta1-16b253de","iteration":7,"namespace":"default","es_name":"elasticsearch-sample"} {"level":"info","@timestamp":"2019-11-18T13:36:46.891Z","logger":"elasticsearch-controller","message":"Ending reconciliation run","ver":"1.0.0-beta1-16b253de","iteration":7,"namespace":"default","name":"elasticsearch-sample","took":0.895109087}with the pr:
{"level":"info","@timestamp":"2019-11-18T13:46:14.841Z","logger":"elasticsearch-controller","message":"Starting reconciliation run","ver":"1.0.0-beta1-7c26b998","iteration":30,"namespace":"default","name":"elasticsearch-sample"} {"level":"info","@timestamp":"2019-11-18T13:46:15.501Z","logger":"zen2","message":"Ensuring no voting exclusions are set","ver":"1.0.0-beta1-7c26b998","namespace":"default","es_name":"elasticsearch-sample"} {"level":"info","@timestamp":"2019-11-18T13:46:15.549Z","logger":"driver","message":"Nodes can't be upgraded","ver":"1.0.0-beta1-7c26b998","namespace":"default","es_name":"elasticsearch-sample","failedPredicates":{"do_not_restart_healthy_node_if_not_green":["elasticsearch-sample-es-default-7","elasticsearch-sample-es-default-6","elasticsearch-sample-es-default-5","elasticsearch-sample-es-default-4","elasticsearch-sample-es-default-3","elasticsearch-sample-es-default-2","elasticsearch-sample-es-default-1","elasticsearch-sample-es-default-0"]}} {"level":"info","@timestamp":"2019-11-18T13:46:15.552Z","logger":"elasticsearch-controller","message":"Updating status","ver":"1.0.0-beta1-7c26b998","iteration":30,"namespace":"default","es_name":"elasticsearch-sample"} {"level":"info","@timestamp":"2019-11-18T13:46:15.552Z","logger":"elasticsearch-controller","message":"Ending reconciliation run","ver":"1.0.0-beta1-7c26b998","iteration":30,"namespace":"default","name":"elasticsearch-sample","took":0.711200508}