Crash when reloading keepalived on startup

[steven-sheehy]

Sometimes on startup, kube-keepalived-vip will crash repeatedly with the below error. It eventually restarts enough and resolves itself.

$ kubectl logs -f keepalived-d5nvd
I1218 20:50:24.964729       9 main.go:237] cleaning ipvs configuration
I1218 20:50:24.967413       9 main.go:191] Creating API server client for https://10.96.0.1:443
I1218 20:50:25.062359       9 main.go:139] starting LVS configuration
I1218 20:50:25.563241       9 main.go:391] No interface was provided, proceeding with the node's default: ens192
W1218 20:50:25.768104       9 main.go:90] requeuing production/dashboard-public, err unexpected error searching configmap production/keepalived: configmap keepalived was not found
W1218 20:50:27.861468       9 main.go:90] requeuing production/edge-mongodb, err unexpected error searching configmap production/keepalived: configmap keepalived was not found
W1218 20:50:30.661474       9 main.go:90] requeuing production/edge-nfacctd, err unexpected error searching configmap production/keepalived: configmap keepalived was not found
W1218 20:50:31.862854       9 main.go:90] requeuing production/prometheus-public, err unexpected error searching configmap production/keepalived: configmap keepalived was not found
I1218 20:50:31.864707       9 main.go:322] starting keepalived to announce VIPs
I1218 20:50:31.961501       9 main.go:315] Starting HTTP server on port 8080
Tue Dec 18 20:50:32 2018: Starting Keepalived v2.0.10 (unknown)
Tue Dec 18 20:50:32 2018: WARNING - keepalived was build for newer Linux 4.18.20, running on Linux 4.15.0-29-generic #31-Ubuntu SMP Tue Jul 17 15:39:52 UTC 2018
Tue Dec 18 20:50:32 2018: Command line: 'keepalived' '--dont-fork' '--log-console' '--release-vips' '--log-detail' '--pid'
Tue Dec 18 20:50:32 2018:               '/keepalived.pid'
Tue Dec 18 20:50:32 2018: Opening file '/etc/keepalived/keepalived.conf'.
I1218 20:50:33.988217       9 keepalived.go:227] Cleanup: [10.0.22.190]
I1218 20:50:33.988255       9 keepalived.go:249] removing configured VIP 10.0.22.190
I1218 20:50:35.161997       9 keepalived.go:173] reloading keepalived
F1218 20:50:35.366135       9 keepalived.go:161] keepalived error: signal: hangup

Sometimes keepalived logs don't even appear before crashing:

I1218 20:50:02.565732       9 main.go:237] cleaning ipvs configuration
I1218 20:50:02.663496       9 main.go:191] Creating API server client for https://10.96.0.1:443
I1218 20:50:02.782646       9 main.go:139] starting LVS configuration
I1218 20:50:04.669568       9 main.go:391] No interface was provided, proceeding with the node's default: ens192
W1218 20:50:04.970123       9 main.go:90] requeuing kube-system/kube-scheduler, err unexpected error searching configmap production/keepalived: configmap keepalived was not found
W1218 20:50:07.065743       9 main.go:90] requeuing production/dashboard, err unexpected error searching configmap production/keepalived: configmap keepalived was not found
W1218 20:50:09.163083       9 main.go:90] requeuing production/metrics-server, err unexpected error searching configmap production/keepalived: configmap keepalived was not found
W1218 20:50:11.063813       9 main.go:90] requeuing production/prometheus, err unexpected error searching configmap production/keepalived: configmap keepalived was not found
I1218 20:50:11.462711       9 main.go:322] starting keepalived to announce VIPs
I1218 20:50:11.462827       9 main.go:315] Starting HTTP server on port 8080
I1218 20:50:12.971297       9 keepalived.go:227] Cleanup: [10.0.22.190]
I1218 20:50:12.971345       9 keepalived.go:249] removing configured VIP 10.0.22.190
I1218 20:50:13.305992       9 keepalived.go:173] reloading keepalived
F1218 20:50:13.306630       9 keepalived.go:161] keepalived error: signal: hangup

I'm not sure exactly how to trigger this, but it looks to happen when keepalived is reloaded (SIGHUP) and that causes the below code to return and exit the program. Perhaps some race condition where keepalived is not fully up to respond to SIGHUP before a reload is triggered?

if err := k.cmd.Wait(); err != nil {
	glog.Fatalf("keepalived error: %v", err)
}

@aledbf Any suggestions on how to address this? Should we put the keepalived start in an another thread or maybe check the err and re-start if hangup?

[steven-sheehy]

Seems like others have had this problem as well.