Investigate the constant pipeline failures

[git-hyagi]

Verify if the API pods crashing is related to a memory leak (https://discourse.pulpproject.org/t/api-server-memory-leak/851/12).

Some ideas to help with the investigation:

• deploy a local minikube (with metrics server) or crc (with monitoring enabled), run the same set of tests we do in our pipeline, and check the API pods' resources utilization
• we can also try to get the logs from the previous instance of the container from API pods (kubectl logs -p) in the show_logs.sh script to see if we can get more information
  https://github.com/pulp/pulp-operator/blob/main/.github/workflows/scripts/show_logs.sh#L30

[git-hyagi]

Actually, the API pods were not crashing but getting restarted because of liveness probe failures.
After doing some tests and modifying the show_logs.sh script from CI I could see:

    Warning  Unhealthy  4m15s (x5 over 5m35s)  kubelet  Liveness probe failed: Get "http://10.244.0.9:24817/pulp/api/v3/status/": dial tcp 10.244.0.9:24817: connect: connection refused

from kubectl describe pod <api pod>:

Containers:
    api:
      Container ID:  docker://49178ba159378cb6f76e9f4f8078d1058c7a71029902fdf946746fd37775263e
      Image:         quay.io/pulp/pulp-minimal:nightly
      Image ID:      docker-pullable://quay.io/pulp/pulp-minimal@sha256:16f6c061991fa1418223ddf87fc35c9a1dbb1b1ab74a9d983c8798ee09f759df
      Port:          24817/TCP
      Host Port:     0/TCP
      Args:
        pulp-api
      State:          Running
        Started:      Mon, 03 Jul 2023 18:50:08 +0000
      Last State:     Terminated
        Reason:       Error    <------- NOT OOM
        Exit Code:    137
        Started:      Mon, 03 Jul 2023 18:46:09 +0000
        Finished:     Mon, 03 Jul 2023 18:50:07 +0000

Checking the logs from a restarted pod, it was running migration tasks when the container got terminated.
As a workaround, we can increase the default failure threshold for the liveness probe while #991 is not implemented.