Handle certificate rotation

[flavio]

Description

Our controller uses two different CA roots:

1. the one generated by cert-manager, which generates the CRD webhook leaf certificate
2. The one generated by the PolicyServerController reconciliation loop the first time a policy server is created, which is used to generate the policy servers' TLS certificate.

There are a few problems with this approach:

1. it creates a dependency on cert-manager, see: Feature Request: make cert-manager optional #422
2. we don't rotate policy server certificates, therefore they will eventually expire.
3. both cert-manager and our implementation on the PolicyServer side do not rotate the CA root (see Allow CA issuer secret rotation cert-manager/cert-manager#2478)

Solution

We need to implement our certificate rotation logic inside the controller.
Specifically, we need to:

• Generate a root CA and use it to configure the webhooks (CRD and policy validating/mutating webhooks)
• Generate the CRD webhook TLS certificate to configure the controller webhook server.
• Generate the TLS policy server certificates every time a new policy server is created
• Implement a CertController that periodically checks if the certificates are expired and regenerates them using a lookahead interval. If a TLS certificate is expired we can re-generate it, and it will be picked up by the servers eventually. If the CA root is expired we need to regenerate all the leaf certificates and implement the logic needed to prevent downtime during this phase.

Tasks

• Bootstrap the root CA and the controller webhook server TLS certificate and remove the cert-manager dependency. #820 feat: cert rotator helm-charts#488
• Rotate TLS certificates #819 feat: certificate rotation #829
• Rotate CA root #818
• tilt: webhooks should have the certificates set #854
• remove cert-manager from the kustomize configurations. #861
• Remove cert-manager from helm-charts #860
• Remove old webhook secret when updating (helm post-update hook) helm-charts#512
• Update certificate handling RFC rfc#39
• Document changes to cert rotation docs#252
• Remove cert-manager requirement step for Kubewarden version 1.17 rancher/kubewarden-ui#839
• Add E2E test for upgrade case, cert rotation #864
• use same certificate generation settings as helm template functions #862
• Testing: add controller webhooks unit test helm-charts#516

[flavio]

Worth a look, something we could leverage: https://github.com/open-policy-agent/cert-controller

[kkaempf]

I always wonder why every project (like kucero for CaaSP) has to re-implement certificate handling. This should be built into Kubernetes.

[fabriziosestito]

Worth a look, something we could leverage: https://github.com/open-policy-agent/cert-controller

We decided not to leverage https://github.com/open-policy-agent/cert-controller since it does not support zero-downtime CA rotation.
Also, we will need to fork it to adapt it to our use case, as we configure policy webhooks dynamically.
Finally, it does not fully support leader election due to this bug.

[flavio]

Closing, all the mandatory tasks have been done.