Deploying example not working #24

iakat · 2020-11-10T23:13:56Z

Steps to reproduce:
DigitalOcean Kubernetes v1.19.3

Apply kustomization.yaml

bases:
  - github.com/amaizfinance/redis-operator/deploy?ref=master

Apply example/k8s_v1alpha1_redis_cr.yaml
No master gets elected, no election happens

Output:

(⎈ |do-nyc3-automuteus:automuteus-redis-ha)~/g/automuteus-infra » k get pod,redis                                      master ✗
NAME                  READY   STATUS    RESTARTS   AGE
pod/redis-example-0   2/2     Running   0          5m2s
pod/redis-example-1   2/2     Running   0          4m43s
pod/redis-example-2   2/2     Running   0          4m27s

NAME                          MASTER   REPLICAS   DESIRED   AGE
redis.k8s.amaiz.com/example                       3         5m2s

Logs operator :

redis-operator-5675f8ccd-gszsm redis-operator {"level":"info","ts":"2020-11-10T23:04:16.409Z","logger":"controller_redis","msg":"Applied *v1.Service","Namespace":"automuteus-redis-ha","Redis":"example"}
redis-operator-5675f8ccd-gszsm redis-operator {"level":"info","ts":"2020-11-10T23:04:16.422Z","logger":"controller_redis","msg":"Applied *v1.Service","Namespace":"automuteus-redis-ha","Redis":"example"}
redis-operator-5675f8ccd-gszsm redis-operator {"level":"info","ts":"2020-11-10T23:04:16.433Z","logger":"controller_redis","msg":"Applied *v1.Service","Namespace":"automuteus-redis-ha","Redis":"example"}
redis-operator-5675f8ccd-gszsm redis-operator {"level":"info","ts":"2020-11-10T23:04:16.446Z","logger":"controller_redis","msg":"Applied *v1.Service","Namespace":"automuteus-redis-ha","Redis":"example"}
redis-operator-5675f8ccd-gszsm redis-operator {"level":"info","ts":"2020-11-10T23:04:16.452Z","logger":"controller_redis","msg":"Applied *v1.ConfigMap","Namespace":"automuteus-redis-ha","Redis":"example"}
redis-operator-5675f8ccd-gszsm redis-operator {"level":"info","ts":"2020-11-10T23:04:16.463Z","logger":"controller_redis","msg":"Applied *v1beta1.PodDisruptionBudget","Namespace":"automuteus-redis-ha","Redis":"example"}
redis-operator-5675f8ccd-gszsm redis-operator {"level":"info","ts":"2020-11-10T23:04:16.495Z","logger":"controller_redis","msg":"Applied *v1.StatefulSet","Namespace":"automuteus-redis-ha","Redis":"example"}
redis-operator-5675f8ccd-gszsm redis-operator {"level":"info","ts":"2020-11-10T23:04:16.597Z","logger":"controller_redis","msg":"Error creating Redis replication, requeue","Namespace":"automuteus-redis-ha","Redis":"example","error":"minimum replication size is not met, only 0 are healthy"}
redis-operator-5675f8ccd-gszsm redis-operator {"level":"info","ts":"2020-11-10T23:04:16.615Z","logger":"controller_redis","msg":"Error creating Redis replication, requeue","Namespace":"automuteus-redis-ha","Redis":"example","error":"minimum replication size is not met, only 0 are healthy"}
redis-operator-5675f8ccd-gszsm redis-operator {"level":"info","ts":"2020-11-10T23:04:16.624Z","logger":"controller_redis","msg":"Error creating Redis replication, requeue","Namespace":"automuteus-redis-ha","Redis":"example","error":"minimum replication size is not met, only 0 are healthy"}
redis-operator-5675f8ccd-gszsm redis-operator {"level":"info","ts":"2020-11-10T23:04:16.740Z","logger":"controller_redis","msg":"Error creating Redis replication, requeue","Namespace":"automuteus-redis-ha","Redis":"example","error":"minimum replication size is not met, only 0 are healthy"}
redis-operator-5675f8ccd-gszsm redis-operator {"level":"info","ts":"2020-11-10T23:04:16.744Z","logger":"controller_redis","msg":"Error creating Redis replication, requeue","Namespace":"automuteus-redis-ha","Redis":"example","error":"minimum replication size is not met, only 0 are healthy"}
redis-operator-5675f8ccd-gszsm redis-operator {"level":"info","ts":"2020-11-10T23:04:19.192Z","logger":"controller_redis","msg":"Error creating Redis replication, requeue","Namespace":"automuteus-redis-ha","Redis":"example","error":"minimum replication size is not met, only 0 are healthy"}
redis-operator-5675f8ccd-gszsm redis-operator {"level":"info","ts":"2020-11-10T23:04:35.778Z","logger":"controller_redis","msg":"Error creating Redis replication, requeue","Namespace":"automuteus-redis-ha","Redis":"example","error":"minimum replication size is not met, only 1 are healthy"}
redis-operator-5675f8ccd-gszsm redis-operator {"level":"info","ts":"2020-11-10T23:04:35.811Z","logger":"controller_redis","msg":"Error creating Redis replication, requeue","Namespace":"automuteus-redis-ha","Redis":"example","error":"minimum replication size is not met, only 1 are healthy"}
redis-operator-5675f8ccd-gszsm redis-operator {"level":"info","ts":"2020-11-10T23:04:35.830Z","logger":"controller_redis","msg":"Error creating Redis replication, requeue","Namespace":"automuteus-redis-ha","Redis":"example","error":"minimum replication size is not met, only 1 are healthy"}
redis-operator-5675f8ccd-gszsm redis-operator {"level":"info","ts":"2020-11-10T23:04:39.674Z","logger":"controller_redis","msg":"Error creating Redis replication, requeue","Namespace":"automuteus-redis-ha","Redis":"example","error":"minimum replication size is not met, only 1 are healthy"}
redis-operator-5675f8ccd-gszsm redis-operator {"level":"error","ts":"2020-11-10T23:04:53.317Z","logger":"kubebuilder.controller","msg":"Reconciler error","controller":"redis-controller","request":"automuteus-redis-ha/example","error":"failed to update Pods: Pod \"redis-example-1\" is invalid: spec: Forbidden: pod updates may not change fields other than `spec.containers[*].image`, `spec.initContainers[*].image`, `spec.activeDeadlineSeconds` or `spec.tolerations` (only additions to existing tolerations)\n  core.PodSpec{\n  \t... // 10 identical fields\n  \tAutomountServiceAccountToken: nil,\n  \tNodeName:                     \"pool-kf1t8lj0w-3kj1e\",\n  \tSecurityContext: &core.PodSecurityContext{\n  \t\t... // 11 identical fields\n  \t\tFSGroupChangePolicy: nil,\n  \t\tSysctls:             nil,\n- \t\tSeccompProfile:      nil,\n+ \t\tSeccompProfile:      &core.SeccompProfile{Type: \"RuntimeDefault\"},\n  \t},\n  \tImagePullSecrets: nil,\n  \tHostname:         \"redis-example-1\",\n  \t... // 15 identical fields\n  }\n; Pod \"redis-example-0\" is invalid: spec: Forbidden: pod updates may not change fields other than `spec.containers[*].image`, `spec.initContainers[*].image`, `spec.activeDeadlineSeconds` or `spec.tolerations` (only additions to existing tolerations)\n  core.PodSpec{\n  \t... // 10 identical fields\n  \tAutomountServiceAccountToken: nil,\n  \tNodeName:                     \"pool-kf1t8lj0w-3kj1g\",\n  \tSecurityContext: &core.PodSecurityContext{\n  \t\t... // 11 identical fields\n  \t\tFSGroupChangePolicy: nil,\n  \t\tSysctls:             nil,\n- \t\tSeccompProfile:      nil,\n+ \t\tSeccompProfile:      &core.SeccompProfile{Type: \"RuntimeDefault\"},\n  \t},\n  \tImagePullSecrets: nil,\n  \tHostname:         \"redis-example-0\",\n  \t... // 15 identical fields\n  }\n;","stacktrace":"github.com/amaizfinance/redis-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\tvendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/amaizfinance/redis-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tvendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\ngithub.com/amaizfinance/redis-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\tvendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/amaizfinance/redis-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\tvendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/amaizfinance/redis-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\tvendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/amaizfinance/redis-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\tvendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
redis-operator-5675f8ccd-gszsm redis-operator {"level":"error","ts":"2020-11-10T23:04:54.371Z","logger":"kubebuilder.controller","msg":"Reconciler error","controller":"redis-controller","request":"automuteus-redis-ha/example","error":"failed to update Pods: Pod \"redis-example-2\" is invalid: spec: Forbidden: pod updates may not change fields other than `spec.containers[*].image`, `spec.initContainers[*].image`, `spec.activeDeadlineSeconds` or `spec.tolerations` (only additions to existing tolerations)\n  core.PodSpec{\n  \t... // 10 identical fields\n  \tAutomountServiceAccountToken: nil,\n  \tNodeName:                     \"pool-kf1t8lj0w-3kj1g\",\n  \tSecurityContext: &core.PodSecurityContext{\n  \t\t... // 11 identical fields\n  \t\tFSGroupChangePolicy: nil,\n  \t\tSysctls:             nil,\n- \t\tSeccompProfile:      nil,\n+ \t\tSeccompProfile:      &core.SeccompProfile{Type: \"RuntimeDefault\"},\n  \t},\n  \tImagePullSecrets: nil,\n  \tHostname:         \"redis-example-2\",\n  \t... // 15 identical fields\n  }\n; Pod \"redis-example-1\" is invalid: spec: Forbidden: pod updates may not change fields other than `spec.containers[*].image`, `spec.initContainers[*].image`, `spec.activeDeadlineSeconds` or `spec.tolerations` (only additions to existing tolerations)\n  core.PodSpec{\n  \t... // 10 identical fields\n  \tAutomountServiceAccountToken: nil,\n  \tNodeName:                     \"pool-kf1t8lj0w-3kj1e\",\n  \tSecurityContext: &core.PodSecurityContext{\n  \t\t... // 11 identical fields\n  \t\tFSGroupChangePolicy: nil,\n  \t\tSysctls:             nil,\n- \t\tSeccompProfile:      nil,\n+ \t\tSeccompProfile:      &core.SeccompProfile{Type: \"RuntimeDefault\"},\n  \t},\n  \tImagePullSecrets: nil,\n  \tHostname:         \"redis-example-1\",\n  \t... // 15 identical fields\n  }\n; Pod \"redis-example-0\" is invalid: spec: Forbidden: pod updates may not change fields other than `spec.containers[*].image`, `spec.initContainers[*].image`, `spec.activeDeadlineSeconds` or `spec.tolerations` (only additions to existing tolerations)\n  core.PodSpec{\n  \t... // 10 identical fields\n  \tAutomountServiceAccountToken: nil,\n  \tNodeName:                     \"pool-kf1t8lj0w-3kj1g\",\n  \tSecurityContext: &core.PodSecurityContext{\n  \t\t... // 11 identical fields\n  \t\tFSGroupChangePolicy: nil,\n  \t\tSysctls:             nil,\n- \t\tSeccompProfile:      nil,\n+ \t\tSeccompProfile:      &core.SeccompProfile{Type: \"RuntimeDefault\"},\n  \t},\n  \tImagePullSecrets: nil,\n  \tHostname:         \"redis-example-0\",\n  \t... // 15 identical fields\n  }\n;","stacktrace":"github.com/amaizfinance/redis-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\tvendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/amaizfinance/redis-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tvendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\ngithub.com/amaizfinance/redis-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\tvendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/amaizfinance/redis-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\tvendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/amaizfinance/redis-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\tvendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/amaizfinance/redis-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\tvendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
redis-operator-5675f8ccd-gszsm redis-operator {"level":"error","ts":"2020-11-10T23:05:12.267Z","logger":"kubebuilder.controller","msg":"Reconciler error","controller":"redis-controller","request":"automuteus-redis-ha/example","error":"failed to update Pods: Pod \"redis-example-0\" is invalid: spec: Forbidden: pod updates may not change fields other than `spec.containers[*].image`, `spec.initContainers[*].image`, `spec.activeDeadlineSeconds` or `spec.tolerations` (only additions to existing tolerations)\n  core.PodSpec{\n  \t... // 10 identical fields\n  \tAutomountServiceAccountToken: nil,\n  \tNodeName:                     \"pool-kf1t8lj0w-3kj1g\",\n  \tSecurityContext: &core.PodSecurityContext{\n  \t\t... // 11 identical fields\n  \t\tFSGroupChangePolicy: nil,\n  \t\tSysctls:             nil,\n- \t\tSeccompProfile:      nil,\n+ \t\tSeccompProfile:      &core.SeccompProfile{Type: \"RuntimeDefault\"},\n  \t},\n  \tImagePullSecrets: nil,\n  \tHostname:         \"redis-example-0\",\n  \t... // 15 identical fields\n  }\n; Pod \"redis-example-2\" is invalid: spec: Forbidden: pod updates may not change fields other than `spec.containers[*].image`, `spec.initContainers[*].image`, `spec.activeDeadlineSeconds` or `spec.tolerations` (only additions to existing tolerations)\n  core.PodSpec{\n  \t... // 10 identical fields\n  \tAutomountServiceAccountToken: nil,\n  \tNodeName:                     \"pool-kf1t8lj0w-3kj1g\",\n  \tSecurityContext: &core.PodSecurityContext{\n  \t\t... // 11 identical fields\n  \t\tFSGroupChangePolicy: nil,\n  \t\tSysctls:             nil,\n- \t\tSeccompProfile:      nil,\n+ \t\tSeccompProfile:      &core.SeccompProfile{Type: \"RuntimeDefault\"},\n  \t},\n  \tImagePullSecrets: nil,\n  \tHostname:         \"redis-example-2\",\n  \t... // 15 identical fields\n  }\n; Pod \"redis-example-1\" is invalid: spec: Forbidden: pod updates may not change fields other than `spec.containers[*].image`, `spec.initContainers[*].image`, `spec.activeDeadlineSeconds` or `spec.tolerations` (only additions to existing tolerations)\n  core.PodSpec{\n  \t... // 10 identical fields\n  \tAutomountServiceAccountToken: nil,\n  \tNodeName:                     \"pool-kf1t8lj0w-3kj1e\",\n  \tSecurityContext: &core.PodSecurityContext{\n  \t\t... // 11 identical fields\n  \t\tFSGroupChangePolicy: nil,\n  \t\tSysctls:             nil,\n- \t\tSeccompProfile:      nil,\n+ \t\tSeccompProfile:      &core.SeccompProfile{Type: \"RuntimeDefault\"},\n  \t},\n  \tImagePullSecrets: nil,\n  \tHostname:         \"redis-example-1\",\n  \t... // 15 identical fields\n  }\n;","stacktrace":"github.com/amaizfinance/redis-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\tvendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/amaizfinance/redis-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tvendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\ngithub.com/amaizfinance/redis-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\tvendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/amaizfinance/redis-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\tvendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/amaizfinance/redis-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\tvendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/amaizfinance/redis-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\tvendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
redis-operator-5675f8ccd-gszsm redis-operator {"level":"error","ts":"2020-11-10T23:05:13.297Z","logger":"kubebuilder.controller","msg":"Reconciler error","controller":"redis-controller","request":"automuteus-redis-ha/example","error":"failed to update Pods: Pod \"redis-example-2\" is invalid: spec: Forbidden: pod updates may not change fields other than `spec.containers[*].image`, `spec.initContainers[*].image`, `spec.activeDeadlineSeconds` or `spec.tolerations` (only additions to existing tolerations)\n  core.PodSpec{\n  \t... // 10 identical fields\n  \tAutomountServiceAccountToken: nil,\n  \tNodeName:                     \"pool-kf1t8lj0w-3kj1g\",\n  \tSecurityContext: &core.PodSecurityContext{\n  \t\t... // 11 identical fields\n  \t\tFSGroupChangePolicy: nil,\n  \t\tSysctls:             nil,\n- \t\tSeccompProfile:      nil,\n+ \t\tSeccompProfile:      &core.SeccompProfile{Type: \"RuntimeDefault\"},\n  \t},\n  \tImagePullSecrets: nil,\n  \tHostname:         \"redis-example-2\",\n  \t... // 15 identical fields\n  }\n; Pod \"redis-example-0\" is invalid: spec: Forbidden: pod updates may not change fields other than `spec.containers[*].image`, `spec.initContainers[*].image`, `spec.activeDeadlineSeconds` or `spec.tolerations` (only additions to existing tolerations)\n  core.PodSpec{\n  \t... // 10 identical fields\n  \tAutomountServiceAccountToken: nil,\n  \tNodeName:                     \"pool-kf1t8lj0w-3kj1g\",\n  \tSecurityContext: &core.PodSecurityContext{\n  \t\t... // 11 identical fields\n  \t\tFSGroupChangePolicy: nil,\n  \t\tSysctls:             nil,\n- \t\tSeccompProfile:      nil,\n+ \t\tSeccompProfile:      &core.SeccompProfile{Type: \"RuntimeDefault\"},\n  \t},\n  \tImagePullSecrets: nil,\n  \tHostname:         \"redis-example-0\",\n  \t... // 15 identical fields\n  }\n; Pod \"redis-example-1\" is invalid: spec: Forbidden: pod updates may not change fields other than `spec.containers[*].image`, `spec.initContainers[*].image`, `spec.activeDeadlineSeconds` or `spec.tolerations` (only additions to existing tolerations)\n  core.PodSpec{\n  \t... // 10 identical fields\n  \tAutomountServiceAccountToken: nil,\n  \tNodeName:                     \"pool-kf1t8lj0w-3kj1e\",\n  \tSecurityContext: &core.PodSecurityContext{\n  \t\t... // 11 identical fields\n  \t\tFSGroupChangePolicy: nil,\n  \t\tSysctls:             nil,\n- \t\tSeccompProfile:      nil,\n+ \t\tSeccompProfile:      &core.SeccompProfile{Type: \"RuntimeDefault\"},\n  \t},\n  \tImagePullSecrets: nil,\n  \tHostname:         \"redis-example-1\",\n  \t... // 15 identical fields\n  }\n;","stacktrace":"github.com/amaizfinance/redis-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\tvendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/amaizfinance/redis-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tvendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\ngithub.com/amaizfinance/redis-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\tvendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/amaizfinance/redis-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\tvendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/amaizfinance/redis-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\tvendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/amaizfinance/redis-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\tvendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}

Redis logs

redis-example-1 redis 1:C 10 Nov 2020 23:04:40.264 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
redis-example-1 redis 1:C 10 Nov 2020 23:04:40.264 # Redis version=6.0.9, bits=64, commit=00000000, modified=0, pid=1, just started
redis-example-1 redis 1:C 10 Nov 2020 23:04:40.264 # Configuration loaded
redis-example-1 exporter time="2020-11-10T23:04:44Z" level=info msg="Redis Metrics Exporter v1.11.1    build date: 2020-08-28-17:21:19    sha1: 3d94cd439e70d3ab478bfa65c1f131ab978a60ad    Go: go1.15    GOOS: linux    GOARCH: amd64"
redis-example-1 exporter time="2020-11-10T23:04:44Z" level=info msg="Providing metrics at :9121/metrics"
redis-example-1 redis 1:M 10 Nov 2020 23:04:40.356 * Running mode=standalone, port=6379.
redis-example-1 redis 1:M 10 Nov 2020 23:04:40.357 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
redis-example-1 redis 1:M 10 Nov 2020 23:04:40.357 # Server initialized
redis-example-1 redis 1:M 10 Nov 2020 23:04:40.357 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo madvise > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled (set to 'madvise' or 'never').
redis-example-1 redis 1:M 10 Nov 2020 23:04:40.358 * Ready to accept connections
redis-example-1 redis 1:S 10 Nov 2020 23:04:51.070 * Before turning into a replica, using my own master parameters to synthesize a cached master: I may be able to synchronize with the new master with just a partial transfer.
redis-example-1 redis 1:S 10 Nov 2020 23:04:51.071 * REPLICAOF 10.244.2.40:6379 enabled (user request from 'id=5 addr=10.244.2.63:56970 fd=8 name= age=0 idle=0 flags=x db=0 sub=0 psub=0 multi=2 qbuf=122 qbuf-free=32646 argv-mem=4 obl=27 oll=0 omem=0 tot-mem=61468 events=r cmd=exec user=default')
redis-example-1 redis 1:S 10 Nov 2020 23:04:52.165 * Connecting to MASTER 10.244.2.40:6379
redis-example-1 redis 1:S 10 Nov 2020 23:04:52.166 * MASTER <-> REPLICA sync started
redis-example-1 redis 1:S 10 Nov 2020 23:04:52.252 * Non blocking connect for SYNC fired the event.
redis-example-1 redis 1:S 10 Nov 2020 23:04:52.255 * Master replied to PING, replication can continue...
redis-example-1 redis 1:S 10 Nov 2020 23:04:52.259 * Trying a partial resynchronization (request 3c0c7132f4d0a76889d82a5867e8487cd36e56b3:1).
redis-example-1 redis 1:S 10 Nov 2020 23:04:52.265 * Full resync from master: 6d8165b7d4cf8a33431579b0722238f74145b459:0
redis-example-1 redis 1:S 10 Nov 2020 23:04:52.265 * Discarding previously cached master state.
redis-example-1 redis 1:S 10 Nov 2020 23:04:52.322 * MASTER <-> REPLICA sync: receiving 175 bytes from master to disk
redis-example-1 redis 1:S 10 Nov 2020 23:04:52.323 * MASTER <-> REPLICA sync: Flushing old data
redis-example-1 redis 1:S 10 Nov 2020 23:04:52.323 * MASTER <-> REPLICA sync: Loading DB in memory
redis-example-1 redis 1:S 10 Nov 2020 23:04:52.355 * Loading RDB produced by version 6.0.9
redis-example-1 redis 1:S 10 Nov 2020 23:04:52.355 * RDB age 0 seconds
redis-example-1 redis 1:S 10 Nov 2020 23:04:52.355 * RDB memory usage when created 1.85 Mb
redis-example-1 redis 1:S 10 Nov 2020 23:04:52.355 * MASTER <-> REPLICA sync: Finished with success
redis-example-2 redis 1:C 10 Nov 2020 23:04:52.739 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
redis-example-2 redis 1:C 10 Nov 2020 23:04:52.739 # Redis version=6.0.9, bits=64, commit=00000000, modified=0, pid=1, just started
redis-example-2 redis 1:C 10 Nov 2020 23:04:52.739 # Configuration loaded
redis-example-2 redis 1:M 10 Nov 2020 23:04:52.741 * Running mode=standalone, port=6379.
redis-example-2 redis 1:M 10 Nov 2020 23:04:52.827 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
redis-example-2 redis 1:M 10 Nov 2020 23:04:52.827 # Server initialized
redis-example-2 redis 1:M 10 Nov 2020 23:04:52.827 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo madvise > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled (set to 'madvise' or 'never').
redis-example-2 redis-example-2 exporter time="2020-11-10T23:04:53Z" level=info msg="Redis Metrics Exporter v1.11.1    build date: 2020-08-28-17:21:19    sha1: 3d94cd439e70d3ab478bfa65c1f131ab978a60ad    Go: go1.15    GOOS: linux    GOARCH: amd64"
redis-example-2 exporter time="2020-11-10T23:04:53Z" level=info msg="Providing metrics at :9121/metrics"
redis 1:M 10 Nov 2020 23:04:52.828 * Ready to accept connections
redis-example-2 redis 1:S 10 Nov 2020 23:05:12.138 * Before turning into a replica, using my own master parameters to synthesize a cached master: I may be able to synchronize with the new master with just a partial transfer.
redis-example-2 redis 1:S 10 Nov 2020 23:05:12.138 * REPLICAOF 10.244.2.40:6379 enabled (user request from 'id=6 addr=10.244.2.63:44632 fd=8 name= age=0 idle=0 flags=x db=0 sub=0 psub=0 multi=2 qbuf=122 qbuf-free=32646 argv-mem=4 obl=27 oll=0 omem=0 tot-mem=61468 events=r cmd=exec user=default')
redis-example-2 redis 1:S 10 Nov 2020 23:05:12.961 * Connecting to MASTER 10.244.2.40:6379
redis-example-2 redis 1:S 10 Nov 2020 23:05:12.962 * MASTER <-> REPLICA sync started
redis-example-2 redis 1:S 10 Nov 2020 23:05:12.962 * Non blocking connect for SYNC fired the event.
redis-example-2 redis 1:S 10 Nov 2020 23:05:12.962 * Master replied to PING, replication can continue...
redis-example-2 redis 1:S 10 Nov 2020 23:05:12.962 * Trying a partial resynchronization (request c0baa0b78d691bc078e06bf443220fbebe8fe262:1).
redis-example-2 redis 1:S 10 Nov 2020 23:05:13.027 * Full resync from master: 6d8165b7d4cf8a33431579b0722238f74145b459:28
redis-example-2 redis 1:S 10 Nov 2020 23:05:13.027 * Discarding previously cached master state.
redis-example-2 redis 1:S 10 Nov 2020 23:05:13.048 * MASTER <-> REPLICA sync: receiving 175 bytes from master to disk
redis-example-2 redis 1:S 10 Nov 2020 23:05:13.049 * MASTER <-> REPLICA sync: Flushing old data
redis-example-2 redis 1:S 10 Nov 2020 23:05:13.049 * MASTER <-> REPLICA sync: Loading DB in memory
redis-example-2 redis 1:S 10 Nov 2020 23:05:13.055 * Loading RDB produced by version 6.0.9
redis-example-2 redis 1:S 10 Nov 2020 23:05:13.055 * RDB age 1 seconds
redis-example-2 redis 1:S 10 Nov 2020 23:05:13.056 * RDB memory usage when created 1.88 Mb
redis-example-2 redis 1:S 10 Nov 2020 23:05:13.056 * MASTER <-> REPLICA sync: Finished with success
redis-example-0 redis 1:C 10 Nov 2020 23:04:18.631 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
redis-example-0 redis 1:C 10 Nov 2020 23:04:18.631 # Redis version=6.0.9, bits=64, commit=00000000, modified=0, pid=1, just started
redis-example-0 redis 1:C 10 Nov 2020 23:04:18.631 # Configuration loaded
redis-example-0 redis 1:M 10 Nov 2020 23:04:18.633 * Running mode=standalone, port=6379.
redis-example-0 redis 1:M 10 Nov 2020 23:04:18.633 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
redis-example-0 redis-example-0 exporter time="2020-11-10T23:04:19Z" level=info msg="Redis Metrics Exporter v1.11.1    build date: 2020-08-28-17:21:19    sha1: 3d94cd439e70d3ab478bfa65c1f131ab978a60ad    Go: go1.15    GOOS: linux    GOARCH: amd64"
redis 1:M 10 Nov 2020 23:04:18.633 # Server initialized
redis-example-0 redis 1:M 10 Nov 2020 23:04:18.633 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo madvise > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled (set to 'madvise' or 'never').
redis-example-0 redis 1:M 10 Nov 2020 23:04:18.633 * Ready to accept connections
redis-example-0 redis 1:M 10 Nov 2020 23:04:52.282 * Replica 10.244.2.181:6379 asks for synchronization
redis-example-0 redis 1:M 10 Nov 2020 23:04:52.282 * Partial resynchronization not accepted: Replication ID mismatch (Replica asked for '3c0c7132f4d0a76889d82a5867e8487cd36e56b3', my replication IDs are '599151193739e42bb7efd6e30649af575fa5fe60' and '0000000000000000000000000000000000000000')
redis-example-0 redis 1:M 10 Nov 2020 23:04:52.282 * Replication backlog created, my new replication IDs are '6d8165b7d4cf8a33431579b0722238f74145b459' and '0000000000000000000000000000000000000000'
redis-example-0 redis 1:M 10 Nov 2020 23:04:52.282 * Starting BGSAVE for SYNC with target: disk
redis-example-0 redis 1:M 10 Nov 2020 23:04:52.283 * Background saving started by pid 36
redis-example-0 redis 36:C 10 Nov 2020 23:04:52.285 * DB saved on disk
redis-example-0 redis 36:C 10 Nov 2020 23:04:52.285 * RDB: 0 MB of memory used by copy-on-write
redis-example-0 redis 1:M 10 Nov 2020 23:04:52.343 * Background saving terminated with success
redis-example-0 redis 1:M 10 Nov 2020 23:04:52.343 * Synchronization with replica 10.244.2.181:6379 succeeded
redis-example-0 redis 1:M 10 Nov 2020 23:05:12.962 * Replica 10.244.2.41:6379 asks for synchronization
redis-example-0 redis 1:M 10 Nov 2020 23:05:12.962 * Partial resynchronization not accepted: Replication ID mismatch (Replica asked for 'c0baa0b78d691bc078e06bf443220fbebe8fe262', my replication IDs are '6d8165b7d4cf8a33431579b0722238f74145b459' and '0000000000000000000000000000000000000000')
redis-example-0 redis 1:M 10 Nov 2020 23:05:12.962 * Starting BGSAVE for SYNC with target: disk
redis-example-0 redis 1:M 10 Nov 2020 23:05:12.963 * Background saving started by pid 62
redis-example-0 redis 62:C 10 Nov 2020 23:05:12.965 * DB saved on disk
redis-example-0 redis 62:C 10 Nov 2020 23:05:12.966 * RDB: 0 MB of memory used by copy-on-write
redis-example-0 redis 1:M 10 Nov 2020 23:05:13.048 * Background saving terminated with success
redis-example-0 redis 1:M 10 Nov 2020 23:05:13.048 * Synchronization with replica 10.244.2.41:6379 succeeded
redis-example-0 exporter time="2020-11-10T23:04:19Z" level=info msg="Providing metrics at :9121/metrics"

The text was updated successfully, but these errors were encountered:

iakat · 2020-11-10T23:27:54Z

Cannot reproduce on k3s v1.18.9+k3s1. (seems to work as intended). Is 1.19 unsupported somehow?

iakat · 2020-11-12T16:13:54Z

Broken on 1.18.10-do.2.

iakat · 2020-11-12T16:28:02Z

Works on Scaleway Kapsule v1.19.3

nrvnrvn · 2020-11-15T11:40:11Z

@sim1 thanks for reporting that. I can confirm the issue with k3d and k8s v 1.19.3

clientVersion:
  buildDate: "2020-10-14T12:50:19Z"
  compiler: gc
  gitCommit: 1e11e4a2108024935ecfcb2912226cedeafd99df
  gitTreeState: clean
  gitVersion: v1.19.3
  goVersion: go1.15.2
  major: "1"
  minor: "19"
  platform: darwin/amd64
serverVersion:
  buildDate: "2020-11-13T07:19:02Z"
  compiler: gc
  gitCommit: 0e4fbfefe1dd8734756dfa4f9ab4fc89665cece4
  gitTreeState: clean
  gitVersion: v1.19.3+k3s3
  goVersion: go1.15.2
  major: "1"
  minor: "19"
  platform: linux/amd64

With this version operator fails to update pods' labels:

redis-operator/pkg/controller/redis/redis_controller.go

Lines 261 to 305 in 6baac7e

    
           // update Pod labels asynchronously and fetch the master Pod's name 
        
           var wg sync.WaitGroup 
        
           errChan := make(chan error, len(podList.Items)) 
        
           masterChan := make(chan string, 1) 
        
           wg.Add(len(podList.Items)) 
        
           for i := range podList.Items { 
        
           	go func(pod corev1.Pod, masterAddress string, wg *sync.WaitGroup) { 
        
           		defer wg.Done() 
        
           		if pod.Status.PodIP == masterAddress { 
        
           			select { 
        
           			case masterChan <- pod.Name: 
        
           				if pod.Labels[roleLabelKey] == masterLabel { 
        
           					return 
        
           				} 
        
           				pod.Labels[roleLabelKey] = masterLabel 
        
           			default: 
        
           				// very unlikely to happen but still... 
        
           				errChan <- fmt.Errorf("IP address conflict for pod %s: %s", pod.Name, pod.Status.PodIP) 
        
           				return 
        
           			} 
        
           		} else { 
        
           			if pod.Labels[roleLabelKey] == replicaLabel { 
        
           				return 
        
           			} 
        
           			pod.Labels[roleLabelKey] = replicaLabel 
        
           		} 
        
           		if err := reconciler.client.Update(ctx, &pod); err != nil { 
        
           			errChan <- err 
        
           		} 
        
           	}(podList.Items[i], master.Host, &wg) 
        
           } 
        
           wg.Wait() 
        
           close(errChan) 
        
           if len(errChan) > 0 { 
        
           	var b strings.Builder 
        
           	defer b.Reset() 
        
           	for err := range errChan { 
        
           		if !errors.IsConflict(err) { 
        
           			_, _ = fmt.Fprintf(&b, " %s;", err) 
        
           		} 
        
           	} 
        
           	if b.Len() > 0 { 
        
           		return reconcile.Result{}, fmt.Errorf("failed to update Pods:%s", b.String())

Let me find another way to update labels without trying to update the whole Pod objects.

nrvnrvn · 2020-11-16T19:44:23Z

@sim1 please verify the fix for the latest image.
k -n redis-operator set image deployments/redis-operator redis-operator=amaiz/redis-operator:latest

maeb · 2021-02-23T13:00:05Z

Image amaiz/redis-operator:latest works on kubernetes version v1.20.2.

nrvnrvn closed this as completed in 549d566 Nov 16, 2020

maeb mentioned this issue Feb 23, 2021

Failed to update Pods in Kubernetes 1.19 #28

Closed

iakat mentioned this issue Feb 27, 2021

Pods are healthy, but operator says: minimum replication size is not met, only 0 are healthy #29

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Deploying example not working #24

Deploying example not working #24

iakat commented Nov 10, 2020 •

edited

Loading

iakat commented Nov 10, 2020 •

edited

Loading

iakat commented Nov 12, 2020

iakat commented Nov 12, 2020

nrvnrvn commented Nov 15, 2020

nrvnrvn commented Nov 16, 2020

maeb commented Feb 23, 2021 •

edited

Loading

Deploying example not working #24

Deploying example not working #24

Comments

iakat commented Nov 10, 2020 • edited Loading

Output:

Logs operator :

Redis logs

iakat commented Nov 10, 2020 • edited Loading

iakat commented Nov 12, 2020

iakat commented Nov 12, 2020

nrvnrvn commented Nov 15, 2020

nrvnrvn commented Nov 16, 2020

maeb commented Feb 23, 2021 • edited Loading

iakat commented Nov 10, 2020 •

edited

Loading

iakat commented Nov 10, 2020 •

edited

Loading

maeb commented Feb 23, 2021 •

edited

Loading