Weird operator behaviour

[EsDmitrii]

Hi!
Today I faced this issue:
When operator's pod can't find stateful set of CRD I've created (sentinel+replication), it starting to restart with error

I0913 14:55:11.293556       1 request.go:665] Waited for 1.039270489s due to client-side throttling, not priority and fairness, request: GET:https://10.233.0.1:443/apis/monitoring.coreos.com/v1alpha1?timeout=32s
{"level":"info","ts":1694616912.0478303,"logger":"controller-runtime.metrics","msg":"Metrics server is starting to listen","addr":":8080"}
{"level":"info","ts":1694616912.0481868,"logger":"setup","msg":"starting manager"}
{"level":"info","ts":1694616912.049558,"msg":"Starting server","path":"/metrics","kind":"metrics","addr":"[::]:8080"}
{"level":"info","ts":1694616912.0496323,"msg":"Starting server","kind":"health probe","addr":"[::]:8081"}
I0913 14:55:12.049722       1 leaderelection.go:248] attempting to acquire leader lease infra-redis-operator/6cab913b.redis.opstreelabs.in...
I0913 14:55:30.927467       1 leaderelection.go:258] successfully acquired lease infra-redis-operator/6cab913b.redis.opstreelabs.in
{"level":"info","ts":1694616930.9280236,"logger":"controller.redis","msg":"Starting EventSource","reconciler group":"redis.redis.opstreelabs.in","reconciler kind":"Redis","source":"kind source: *v1beta1.Redis"}
{"level":"info","ts":1694616930.9281197,"logger":"controller.redis","msg":"Starting Controller","reconciler group":"redis.redis.opstreelabs.in","reconciler kind":"Redis"}
{"level":"info","ts":1694616930.9293966,"logger":"controller.redisreplication","msg":"Starting EventSource","reconciler group":"redis.redis.opstreelabs.in","reconciler kind":"RedisReplication","source":"kind source: *v1beta1.RedisReplication"}
{"level":"info","ts":1694616930.929487,"logger":"controller.redisreplication","msg":"Starting Controller","reconciler group":"redis.redis.opstreelabs.in","reconciler kind":"RedisReplication"}
{"level":"info","ts":1694616930.9304016,"logger":"controller.rediscluster","msg":"Starting EventSource","reconciler group":"redis.redis.opstreelabs.in","reconciler kind":"RedisCluster","source":"kind source: *v1beta1.RedisCluster"}
{"level":"info","ts":1694616930.930462,"logger":"controller.rediscluster","msg":"Starting Controller","reconciler group":"redis.redis.opstreelabs.in","reconciler kind":"RedisCluster"}
{"level":"info","ts":1694616930.9316761,"logger":"controller.redissentinel","msg":"Starting EventSource","reconciler group":"redis.redis.opstreelabs.in","reconciler kind":"RedisSentinel","source":"kind source: *v1beta1.RedisSentinel"}
{"level":"info","ts":1694616930.9317424,"logger":"controller.redissentinel","msg":"Starting Controller","reconciler group":"redis.redis.opstreelabs.in","reconciler kind":"RedisSentinel"}
{"level":"info","ts":1694616931.0295331,"logger":"controller.redis","msg":"Starting workers","reconciler group":"redis.redis.opstreelabs.in","reconciler kind":"Redis","worker count":1}
{"level":"info","ts":1694616931.0304148,"logger":"controller.redisreplication","msg":"Starting workers","reconciler group":"redis.redis.opstreelabs.in","reconciler kind":"RedisReplication","worker count":1}
{"level":"info","ts":1694616931.0305955,"logger":"controllers.RedisReplication","msg":"Reconciling opstree redis replication controller","Request.Namespace":"infra-harbor","Request.Name":"harbor-redis"}
{"level":"info","ts":1694616931.0318134,"logger":"controller.rediscluster","msg":"Starting workers","reconciler group":"redis.redis.opstreelabs.in","reconciler kind":"RedisCluster","worker count":1}
{"level":"info","ts":1694616931.03222,"logger":"controller.redissentinel","msg":"Starting workers","reconciler group":"redis.redis.opstreelabs.in","reconciler kind":"RedisSentinel","worker count":1}
{"level":"info","ts":1694616931.0324035,"logger":"controllers.RedisSentinel","msg":"Reconciling opstree redis controller","Request.Namespace":"infra-harbor","Request.Name":"harbor-redis"}
{"level":"info","ts":1694616931.0367954,"logger":"controller_redis","msg":"Redis statefulset get action failed","Request.StatefulSet.Namespace":"infra-harbor","Request.StatefulSet.Name":"harbor-redis"}
{"level":"info","ts":1694616931.039696,"logger":"controller_redis","msg":"Successfully Execute the Get Request","Request.RedisManager.Namespace":"infra-harbor","Request.RedisManager.Name":"harbor-redis","replication name":"harbor-redis","namespace":"infra-harbor"}
{"level":"info","ts":1694616931.056547,"logger":"controller_redis","msg":"Redis statefulset get action failed","Request.StatefulSet.Namespace":"infra-harbor","Request.StatefulSet.Name":"harbor-redis"}
{"level":"error","ts":1694616931.0565653,"logger":"controller_redis","msg":"Failed to Get the Statefulset of the","Request.RedisManager.Namespace":"infra-harbor","Request.RedisManager.Name":"harbor-redis","custom resource":"harbor-redis","in namespace":"infra-harbor","error":"statefulsets.apps \"harbor-redis\" not found","stacktrace":"github.com/OT-CONTAINER-KIT/redis-operator/k8sutils.getRedisReplicationMasterIP\n\t/workspace/k8sutils/redis-sentinel.go:310\ngithub.com/OT-CONTAINER-KIT/redis-operator/k8sutils.getSentinelEnvVariable\n\t/workspace/k8sutils/redis-sentinel.go:246\ngithub.com/OT-CONTAINER-KIT/redis-operator/k8sutils.generateRedisSentinelContainerParams\n\t/workspace/k8sutils/redis-sentinel.go:151\ngithub.com/OT-CONTAINER-KIT/redis-operator/k8sutils.RedisSentinelSTS.CreateRedisSentinelSetup\n\t/workspace/k8sutils/redis-sentinel.go:75\ngithub.com/OT-CONTAINER-KIT/redis-operator/k8sutils.CreateRedisSentinel\n\t/workspace/k8sutils/redis-sentinel.go:48\ngithub.com/OT-CONTAINER-KIT/redis-operator/controllers.(*RedisSentinelReconciler).Reconcile\n\t/workspace/controllers/redissentinel_controller.go:50\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227"}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x1425f03]

goroutine 379 [running]:
github.com/OT-CONTAINER-KIT/redis-operator/k8sutils.GetRedisNodesByRole(0xc0007fb8c0, {0x1745a91, 0x6})
	/workspace/k8sutils/redis.go:479 +0x263
github.com/OT-CONTAINER-KIT/redis-operator/k8sutils.getRedisReplicationMasterIP(0xc0007fb200)
	/workspace/k8sutils/redis-sentinel.go:310 +0x3f1
github.com/OT-CONTAINER-KIT/redis-operator/k8sutils.getSentinelEnvVariable(0xc0007fb200)
	/workspace/k8sutils/redis-sentinel.go:246 +0x25
github.com/OT-CONTAINER-KIT/redis-operator/k8sutils.generateRedisSentinelContainerParams(0xc0007fb200, _, _)
	/workspace/k8sutils/redis-sentinel.go:151 +0xa5
github.com/OT-CONTAINER-KIT/redis-operator/k8sutils.RedisSentinelSTS.CreateRedisSentinelSetup({{0x174762e, 0x8}, 0x0, 0x0, 0x0, 0xc0003c3a58, 0xc0003c3a70}, 0xc0007fb200)
	/workspace/k8sutils/redis-sentinel.go:75 +0x4ab
github.com/OT-CONTAINER-KIT/redis-operator/k8sutils.CreateRedisSentinel(0xc0007fb200)
	/workspace/k8sutils/redis-sentinel.go:48 +0xba
github.com/OT-CONTAINER-KIT/redis-operator/controllers.(*RedisSentinelReconciler).Reconcile(0xc0006f2720, {0xc000998180, 0x157a440}, {{{0xc0000497c0, 0x1686160}, {0xc0000497b0, 0x30}}})
	/workspace/controllers/redissentinel_controller.go:50 +0x29c
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0xc00011a2c0, {0x1964f98, 0xc000998180}, {{{0xc0000497c0, 0x1686160}, {0xc0000497b0, 0x413a34}}})
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:114 +0x26f
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc00011a2c0, {0x1964ef0, 0xc0004e0c40}, {0x15d0360, 0xc0009877c0})
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:311 +0x33e
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc00011a2c0, {0x1964ef0, 0xc0004e0c40})
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266 +0x205
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227 +0x85
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:223 +0x357

Is it expected behaviour?
Or what am I doing wrong?
Thank you in advance!

[shaunbugler]

Hi @EsDmitrii , I am wondering if you used redis operator older than 0.14.0, they fixed a crash with go panic in this release, see commit #385

[EsDmitrii]

yep, I used 0.15.3 helm ver

[shaunbugler]

I do see another fix #615 which was added to operator 0.15.1 from my side I an only say try the new version otherwise log a new issue

[EsDmitrii]

got it, thank you