OLM creating tons of packageservers

[mjpitz]

This morning I logged into one of our clusters and saw a lot of instances of packageserver running. I know when I ran this previously, I didn't have nearly this many instances. Is this expected? (I suspect not) Anyone have any idea what's going on here? Seems like something might be misconfigured or not cleaned up properly.

$ kubectl get all -n olm
NAME                                    READY   STATUS    RESTARTS   AGE
pod/catalog-operator-6c6dd66dc9-4ft2h   1/1     Running   1          6d20h
pod/olm-operator-54bfd7d46c-2tkxh       1/1     Running   1          6d20h
pod/olm-operators-pbp2w                 1/1     Running   2          6d20h
pod/operatorhubio-catalog-2nz9j         1/1     Running   2          6d20h
pod/packageserver-546ccb4c8b-cf7hd      1/1     Running   0          22h
pod/packageserver-546ccb4c8b-x7zjj      1/1     Running   0          22h
pod/packageserver-578fc7674f-2wsm6      1/1     Running   1          4d18h
pod/packageserver-5dd864f7fb-lkskn      1/1     Running   0          22h
pod/packageserver-69ddc9975-27pps       1/1     Running   1          22h
pod/packageserver-6d4f79694-lscdk       1/1     Running   1          6d20h
pod/packageserver-6d4f79694-zz2w2       0/1     Running   2          6d20h
pod/packageserver-75f8bf699d-vzvwr      1/1     Running   1          4d18h
pod/packageserver-79b5d4c4dd-mhfrs      1/1     Running   1          4d18h
pod/packageserver-79b5d4c4dd-skrps      1/1     Running   1          4d18h
pod/packageserver-79c79755fc-j7jj5      1/1     Running   1          22h
pod/packageserver-79c79755fc-vtvdq      1/1     Running   0          22h
pod/packageserver-b7fcbb874-gfcnh       1/1     Running   0          22h
pod/packageserver-c9666df5d-8l5rw       1/1     Running   1          4d18h
pod/packageserver-c9666df5d-kg7kl       1/1     Running   1          4d18h

NAME                                       TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)     AGE
service/olm-operators                      ClusterIP   10.43.57.151   <none>        50051/TCP   6d20h
service/operatorhubio-catalog              ClusterIP   10.43.161.67   <none>        50051/TCP   6d20h
service/v1-packages-operators-coreos-com   ClusterIP   10.43.5.173    <none>        443/TCP     21h

NAME                               READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/catalog-operator   1/1     1            1           6d21h
deployment.apps/olm-operator       1/1     1            1           6d21h
deployment.apps/packageserver      14/2    0            14          6d20h

NAME                                          DESIRED   CURRENT   READY   AGE
replicaset.apps/catalog-operator-6c6dd66dc9   1         1         1       6d21h
replicaset.apps/olm-operator-54bfd7d46c       1         1         1       6d21h
replicaset.apps/packageserver-546ccb4c8b      2         2         2       22h
replicaset.apps/packageserver-578fc7674f      0         1         1       4d18h
replicaset.apps/packageserver-5888f88f57      0         0         0       22h
replicaset.apps/packageserver-5899c5bdb5      0         0         0       4d17h
replicaset.apps/packageserver-594b7d6794      0         0         0       2d1h
replicaset.apps/packageserver-5b6d67c548      0         0         0       22h
replicaset.apps/packageserver-5d69678bdf      0         0         0       22h
replicaset.apps/packageserver-5d799bd4c4      0         0         0       21h
replicaset.apps/packageserver-5dd864f7fb      0         1         1       2d1h
replicaset.apps/packageserver-5f8587f4d       0         0         0       2d1h
replicaset.apps/packageserver-5f98d44948      0         0         0       2d1h
replicaset.apps/packageserver-6556744b7f      0         0         0       2d1h
replicaset.apps/packageserver-657d9df547      0         0         0       6d20h
replicaset.apps/packageserver-685c6dc596      0         0         0       2d1h
replicaset.apps/packageserver-689764d7bc      0         0         0       22h
replicaset.apps/packageserver-6898c46457      0         0         0       2d1h
replicaset.apps/packageserver-69c5f985c6      0         0         0       2d1h
replicaset.apps/packageserver-69ddc9975       0         1         1       22h
replicaset.apps/packageserver-6c5f6c654b      0         0         0       22h
replicaset.apps/packageserver-6c6d4c5947      0         0         0       22h
replicaset.apps/packageserver-6d4f79694       0         2         1       6d20h
replicaset.apps/packageserver-6d6c99cc59      0         0         0       6d20h
replicaset.apps/packageserver-6f4b98bb6       0         0         0       22h
replicaset.apps/packageserver-75f8bf699d      0         1         1       4d18h
replicaset.apps/packageserver-77f5b48699      0         0         0       4d17h
replicaset.apps/packageserver-79b5d4c4dd      0         2         2       4d18h
replicaset.apps/packageserver-79c79755fc      1         2         2       22h
replicaset.apps/packageserver-7cffb5bc4       0         0         0       2d1h
replicaset.apps/packageserver-7dcc5f985f      0         0         0       22h
replicaset.apps/packageserver-7dcdc5966       0         0         0       2d1h
replicaset.apps/packageserver-84cd7bb7b4      0         0         0       4d17h
replicaset.apps/packageserver-85b99697f6      0         0         0       2d1h
replicaset.apps/packageserver-b7fcbb874       0         1         1       22h
replicaset.apps/packageserver-c9666df5d       0         2         2       4d18h

[mjpitz]

Upon further review I believe I found the culprit too be one of our internal processes preventing the termination of the older replica sets.