Fix Pebble checks on workers when TLS is enabled

[michaeldmitry]

Issue

With the current, worker pebble checks, if TLS is enabled, the worker unit would be stuck in BlockedStatus(node down) with errors in workload logs indicating that certificate is valid for [coordinator hostname], not for [worker hostname]. The reason is we're passing the worker, a cert requested by the coordinator with sans that don't contain workers' FQDNs.

Solution

• Pass the integrated workers' hostnames (FQDNs) to the cert's SANs and refresh the CSR on cluster_relation_changed_event. This will try to add/remove the worker's SAN when its added/removed to the cluster.
• run update-ca-certificates to workloads to trust the CA.

Context

Addition of refresh_events to CertHandler library canonical/observability-libs#108

Testing Instructions

Deploy S3, SSC

curl https://raw.githubusercontent.com/canonical/tempo-coordinator-k8s-operator/main/scripts/deploy_minio.py | MINIO_MODEL=test python3
juju deploy self-signed-certificates ssc --channel edge --trust

Pack & deploy Tempo coordinator and with this cos-lib

Pack & deploy Tempo worker, with this cos-lib and pass socket.getfqdn() to readiness_check_endpoint

Integrate

jhack imtarix fill

Verify

• Wait until units settle and in active/idle state
• Inspect logs of worker to make sure there are no certificate errors

Scale

juju add-unit tempo-worker

Verify

• Wait until all units settle and in active/idle state
• Inspect logs of both workers to make sure there are no certificate errors

[PietroPasotti]

LGTM, good job finding out the issue in the first place!

[PietroPasotti]

unittests are fixed, but they depend on canonical/observability-libs#108