Certwatcher issue

[elmira-aliyeva]

What steps did you take and what happened:

When I recreate Secret (reapply manifest), data field in Secret is emptied, and after it is not updated with certs data. So if I will then recreate ValidatingWebhookConfiguration, caBundle field will not be updated in the config and I see error:

Error from server (InternalError): error when creating "pod.yaml": Internal error occurred: failed calling webhook "validation.gatekeeper.sh": failed to call webhook: Post "https://gatekeeper-webhook-service.gatekeeper-system.svc:443/v1/admit?timeout=3s": x509: certificate signed by unknown authority

Controller manager logs:

{"level":"info","ts":1671573178.9620776,"logger":"cert-rotation","msg":"Ensuring CA cert","name":"gatekeeper-validating-webhook-configuration","gvk":"admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration","name":"gatekeeper-validating-webhook-configuration","gvk":"admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration"}
{"level":"info","ts":1671573178.9682496,"logger":"cert-rotation","msg":"Ensuring CA cert","name":"gatekeeper-validating-webhook-configuration","gvk":"admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration","name":"gatekeeper-validating-webhook-configuration","gvk":"admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration"}
{"level":"info","ts":1671573180.771452,"logger":"cert-rotation","msg":"certs are ready in /certs"}
{"level":"info","ts":1671573180.7714906,"logger":"cert-rotation","msg":"CA certs are injected to webhooks"}
{"level":"info","ts":1671573180.7740057,"logger":"controller-runtime.certwatcher","msg":"Updated current TLS certificate"}
{"level":"info","ts":1671573180.7808604,"logger":"controller-runtime.certwatcher","msg":"Starting certificate watcher"}
{"level":"error","ts":1671573202.8345468,"logger":"cert-rotation","msg":"secret is not well-formed, cannot update webhook configurations","error":"Cert secret is not well-formed, missing ca.crt","errorVerbose":"Cert secret is not well-formed, missing ca.crt\ngithub.com/open-policy-agent/cert-controller/pkg/rotator.buildArtifactsFromSecret\n\t/go/src/github.com/open-policy-agent/gatekeeper/vendor/github.com/open-policy-agent/cert-controller/pkg/rotator/rotator.go:385\ngithub.com/open-policy-agent/cert-controller/pkg/rotator.(*ReconcileWH).Reconcile\n\t/go/src/github.com/open-policy-agent/gatekeeper/vendor/github.com/open-policy-agent/cert-controller/pkg/rotator/rotator.go:646\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/src/github.com/open-policy-agent/gatekeeper/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/src/github.com/open-policy-agent/gatekeeper/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/open-policy-agent/gatekeeper/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/src/github.com/open-policy-agent/gatekeeper/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594","stacktrace":"github.com/open-policy-agent/cert-controller/pkg/rotator.(*ReconcileWH).Reconcile\n\t/go/src/github.com/open-policy-agent/gatekeeper/vendor/github.com/open-policy-agent/cert-controller/pkg/rotator/rotator.go:648\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/src/github.com/open-policy-agent/gatekeeper/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/src/github.com/open-policy-agent/gatekeeper/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/open-policy-agent/gatekeeper/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/src/github.com/open-policy-agent/gatekeeper/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234"}
{"level":"error","ts":1671573245.1341057,"logger":"controller-runtime.certwatcher","msg":"error re-watching file","error":"no such file or directory","stacktrace":"sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).handleEvent\n\t/go/src/github.com/open-policy-agent/gatekeeper/vendor/sigs.k8s.io/controller-runtime/pkg/certwatcher/certwatcher.go:147\nsigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Watch\n\t/go/src/github.com/open-policy-agent/gatekeeper/vendor/sigs.k8s.io/controller-runtime/pkg/certwatcher/certwatcher.go:103"} 
{"level":"error","ts":1671573245.1341906,"logger":"controller-runtime.certwatcher","msg":"error re-reading certificate","error":"open /certs/tls.crt: no such file or directory","stacktrace":"sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).handleEvent\n\t/go/src/github.com/open-policy-agent/gatekeeper/vendor/sigs.k8s.io/controller-runtime/pkg/certwatcher/certwatcher.go:152\nsigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Watch\n\t/go/src/github.com/open-policy-agent/gatekeeper/vendor/sigs.k8s.io/controller-runtime/pkg/certwatcher/certwatcher.go:103"}

What did you expect to happen:
Certs to be updated in Secret object.

Environment:

• Gatekeeper version: v3.10.0-beta.1
• Kubernetes version: (use kubectl version): v1.24.6

[jimmyraywv]

Any insight into this yet? I am seeing this in GK v3.10.0, minikube (k8s v1.25.3).

[maxsmythe]

It looks like cert watcher only regenerates certs on a timer:

https://github.com/open-policy-agent/cert-controller/blob/fef091abda36d414b732733758294ae036d1242f/pkg/rotator/rotator.go#L215-L231

Updating the following code to call refreshCertsIfNeeded() and then requeue the request could fix the issue:

https://github.com/open-policy-agent/cert-controller/blob/fef091abda36d414b732733758294ae036d1242f/pkg/rotator/rotator.go#L672-L676

In the interim, nudging a pod (e.g. kubectl delete pod for one of the webhook pods) would trigger cert regeneration

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

[maxsmythe]

still an open bug

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

[maxsmythe]

@acpana interested?

[acpana]

thanks for the tag max! I can have a look at this in my downtime from other projects. I will assign it to myself when I get to it. In the meantime, folks can feel free to jump on it if they have cycles.