Panic on invalid cast

[uthark]

CNI Plugin panics

ERROR: logging before flag.Parse: E0708 19:00:50.067643      35 runtime.go:66] Observed a panic: &runtime.TypeAssertionError{_interface:(*runtime._type)(0x16791c0), concrete:(*runtime._type)(0x174dfc0), asserted:(*runtime._type)(0x18dd460), missingMethod:""} (interface conversion: interface {} is cache.DeletedFinalStateUnknown, not *unstructured.Unstructured)
/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/runtime/runtime.go:72
/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/runtime/runtime.go:65
/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/runtime/runtime.go:51
/usr/local/go/src/runtime/panic.go:522
/usr/local/go/src/runtime/iface.go:248
/go/pkg/mod/github.com/operator-framework/operator-sdk@v0.0.7/pkg/sdk/informer.go:123
/go/pkg/mod/k8s.io/client-go@v0.0.0-20180806134042-1f13a808da65/tools/cache/controller.go:209
/go/pkg/mod/k8s.io/client-go@v0.0.0-20180806134042-1f13a808da65/tools/cache/shared_informer.go:556
/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/wait/wait.go:203
/go/pkg/mod/k8s.io/client-go@v0.0.0-20180806134042-1f13a808da65/tools/cache/shared_informer.go:548
/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/wait/wait.go:133
/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/wait/wait.go:134
/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/wait/wait.go:88
/go/pkg/mod/k8s.io/client-go@v0.0.0-20180806134042-1f13a808da65/tools/cache/shared_informer.go:546
/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/wait/wait.go:71
/usr/local/go/src/runtime/asm_amd64.s:1337
panic: interface conversion: interface {} is cache.DeletedFinalStateUnknown, not *unstructured.Unstructured [recovered]
	panic: interface conversion: interface {} is cache.DeletedFinalStateUnknown, not *unstructured.Unstructured

goroutine 200 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/runtime/runtime.go:58 +0x105
panic(0x16d58a0, 0xc010834510)
	/usr/local/go/src/runtime/panic.go:522 +0x1b5
github.com/operator-framework/operator-sdk/pkg/sdk.(*informer).handleDeleteResourceEvent(0xc000164fc0, 0x174dfc0, 0xc000bb32e0)
	/go/pkg/mod/github.com/operator-framework/operator-sdk@v0.0.7/pkg/sdk/informer.go:123 +0x1f9
k8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnDelete(...)
	/go/pkg/mod/k8s.io/client-go@v0.0.0-20180806134042-1f13a808da65/tools/cache/controller.go:209
k8s.io/client-go/tools/cache.(*processorListener).run.func1.1(0x42bb5c, 0xc0005ecde8, 0x0)
	/go/pkg/mod/k8s.io/client-go@v0.0.0-20180806134042-1f13a808da65/tools/cache/shared_informer.go:556 +0x22b
k8s.io/apimachinery/pkg/util/wait.ExponentialBackoff(0x989680, 0x3ff0000000000000, 0x3fb999999999999a, 0x5, 0xc00491be38, 0x42b66f, 0xc00036abc0)
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/wait/wait.go:203 +0xde
k8s.io/client-go/tools/cache.(*processorListener).run.func1()
	/go/pkg/mod/k8s.io/client-go@v0.0.0-20180806134042-1f13a808da65/tools/cache/shared_informer.go:548 +0x89
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc0005ecf68)
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/wait/wait.go:133 +0x54
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00491bf68, 0xdf8475800, 0x0, 0x42bf01, 0xc00053e660)
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/wait/wait.go:134 +0xf8
k8s.io/apimachinery/pkg/util/wait.Until(...)
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/wait/wait.go:88
k8s.io/client-go/tools/cache.(*processorListener).run(0xc0004a6400)
	/go/pkg/mod/k8s.io/client-go@v0.0.0-20180806134042-1f13a808da65/tools/cache/shared_informer.go:546 +0x9c
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc000165090, 0xc00055a880)
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/wait/wait.go:71 +0x4f
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20180621070125-103fd098999d/pkg/util/wait/wait.go:69 +0x62

Used operator framework is v0.0.7, panic is here: https://github.com/operator-framework/operator-sdk/blob/v0.0.7/pkg/sdk/informer.go#L123

New version is 0.8.7.
Probably, upgrading will solve the issue.

[adammw]

Worth noting that this usually occurs immediately after a string of these warnings:

ERROR: logging before flag.Parse: W0708 18:44:17.313687      37 reflector.go:341] pkg/mod/k8s.io/client-go@v0.0.0-20180806134042-1f13a808da65/tools/cache/reflector.go:99: watch of *unstructured.Unstructured ended with: unexpected object: &{map[apiVersion:v1 code:410 kind:Status message:too old resource version: 226782159 (226782788) metadata:map[] reason:Gone status:Failure]}

[uthark]

Additional information: we recently observed it again, while we were upgrading API servers.
As a result, CNI plugin restarts and then tries to restore its state of the world by polling API servers. On bigger clusters, we see immediate increase of the load on API Servers right after start.

[mogren]

Depending on the work being done in #560 or by @jaypipes, we should be able to close this out soon.

[jayanthvn]

Hi @uthark

We are tracking operator-sdk upgrade as part of #745. Closing this issue for now. Kindly let us know if you still facing the issue.

Thank you!