Add check for primary endpoint ready

[marceloneppel]

Issue

• When relating the PostgreSQL k8s charm with other charm before the PostgreSQL charm is fully initialised (in an Active state), we can face some errors related to database connection problems. This can be checked by testing the relation between this charm and the PgBouncer k8s charm.
• Those problems happen due the PostgreSQL charm primary k8s endpoint/service is not yet correctly pointing to the primary pod (it takes some time to work).
• Jira issue: DPE-701

Solution

• Avoid setting the cluster_initialised flag (that is checked in multiple charm events) before the primary endpoint/service is fully ready by checking its accessibility.

Context

• Files to FOCUS when reviewing this PR:

  • src/charm.py: has the call to the check, an update to the upgrade charm event handler to avoid problems withe the primary endpoint being deleted when the pod is killed and also an update to the update status event holder to avoid logging error messages when the PostgreSQL/Patroni pebble service is not ready yet.

  • src/patroni.py: has the implementation of the primary endpoint/service readiness check.

  • src/resources.yaml has an additional port that is used to contact the primary endpoint during the initial check on charm.py pebble ready event.

  • tests/integration/new_relations/test_new_relations.py has an update to merge two tests in one (in a way that makes the PostgreSQL charm to be related faster to the application charm; this avoids any kind of regression on this new implementation).

• Extra details about the other files:

  • tests/unit/test_charm.py and tests/unit/test_patroni.py were updated just to match the new implementations.

Testing

• Unit tests were updated.
• Integration tests for the relation were changed to (the same will work for legacy relations, but the charms that are used on those tests work differently, so they don't trigger the specific issue that is being solved through this PR).
• The changes from this PR were manually tested by deploying this charm, the PgBouncer k8s charm and immediately relating them (which triggers the problem before these changes).

Release Notes

• Add check for started/ready primary endpoint/service.

[MiaAltieri]

Looks awesome!! One minor suggestion

[MiaAltieri]

If the state isn't running is it worth retrying? ie

        try:
            for attempt in Retrying(stop=stop_after_delay(10), wait=wait_fixed(3)):
                with attempt:
                    r = requests.get(
                        f"{'https' if self._tls_enabled else 'http'}://{self._primary_endpoint}:8008/health",
                        verify=self._verify,
                    )
                    if not r.json()["state"] == "running":
                         raise endpointNotReadyError
        except RetryError: 
            return False

        return True

[marceloneppel]

Yes. It is a great improvement! I will implement it. Thanks Mia!

[marceloneppel]

Updated on fbb4332.

[MiaAltieri]

cleverly tested :)

[marceloneppel]

Added an additional test for relation broken on tests/integration/new_relations/test_new_relations.py via 3b61943. The updated library code was ok on the VM charm but it was missing here, so I updated here and added the test. Now the relations are more robust.

[MiaAltieri]

Looks awesome, really good work here :)