Agent crashes when API server is not available

[jiria]

Agent crashes when API server is not available.

Here is an example:

akri.sh Agent start
akri.sh KUBERNETES_PORT found ... env_logger::init
[2021-01-23T05:40:51Z TRACE agent] akri.sh KUBERNETES_PORT found ... env_logger::init finished
[2021-01-23T05:40:51Z INFO  akri_shared::akri::metrics] starting metrics server on port 8080 at /metrics
[2021-01-23T05:40:51Z INFO  agent::util::config_action] do_config_watch - enter
[2021-01-23T05:40:51Z TRACE akri_shared::k8s] Loading in-cluster config
[2021-01-23T05:40:51Z TRACE agent::util::slot_reconciliation] periodic_slot_reconciliation - start
[2021-01-23T05:40:51Z TRACE akri_shared::k8s] Loading in-cluster config
[2021-01-23T05:40:51Z TRACE agent::util::slot_reconciliation] periodic_slot_reconciliation - iteration pre delay_for
[2021-01-23T05:40:51Z TRACE akri_shared::akri::configuration] get_configurations enter
[2021-01-23T05:40:51Z TRACE akri_shared::akri::configuration] get_configurations kube_client.request::<KubeAkriInstanceList>(akri_config_type.list(...)?).await?
[2021-01-23T05:40:52Z TRACE akri_shared::akri::configuration] get_configurations kube_client.request error: ReqwestError(reqwest::Error { kind: Request, url: Url { scheme: "https", host: Some(Ipv4(10.43.0.1)), port: None, path: "/apis/akri.sh/v0/configurations", query: Some(""), fragment: None }, source: hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 113, kind: Other, message: "No route to host" })) })
thread 'tokio-runtime-worker' panicked at 'called `Result::unwrap()` on an `Err` value: ReqwestError(reqwest::Error { kind: Request, url: Url { scheme: "https", host: Some(Ipv4(10.43.0.1)), port: None, path: "/apis/akri.sh/v0/configurations", query: Some(""), fragment: None }, source: hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 113, kind: Other, message: "No route to host" })) })', agent/src/main.rs:68:48
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Error: JoinError::Panic(...)

Consider using the cached configuration instead of crashing.

[jiayihu]

Ideally, the agent should be alive for about 5mins before anything extreme like crashing. 5mins is for instance the standard eviction timeout duration for a node: https://kubernetes.io/docs/concepts/architecture/nodes/#condition

Another idea is to retry a fixed number of times, maybe with something like exponential backoff. Krustlet does something similar when it needs to register itself as new node to the apiserver: https://github.com/deislabs/krustlet/blob/master/crates/kubelet/src/node/mod.rs#L80

[github-actions]

Issue has been automatically marked as stale due to inactivity for 45 days. Update the issue to remove label, otherwise it will be automatically closed.

[adithyaj]

@kate-goldenring did #374 resolve this issue? Should we close it?

[kate-goldenring]

@adithyaj I am not sure we ever tested if it resolved it but i think it is safe to close this and reopen it if someone else experiences it still

[kate-goldenring]

This issue is still not resolved according to #557

[diconico07]

From what I understand of the kube-rs documentation:

Errors from the underlying watch are propagated, after which the stream will go into recovery mode on the next poll.

So my guess is that this construct is at fault:

let watcher = watcher(resource, ListParams::default());
let mut informer = watcher.boxed();
...
while let Some(event) = informer.try_next().await? {
...

When doing this we are exiting the loop in case of an error probably panic, we should check for error within the loop and handle it locally (maybe also have a backoff mechanism).

I'll try to do a PR for this.