[PWX-38584][PWX-38575] Updated Autopilot deployment to include readiness probes

[mdaigavane-px]

What this PR does :

This PR adds Readiness Probes to Autopilot deployments from autopilot versions greater than 1.3.15.

Why we need it

Autopilot relies heavily on Prometheus signals to execute its operations. If Prometheus becomes unreachable, Autopilot's expansion operations will fail.

Users typically become aware of this issue only after an expansion operation fails, which is too late.

To address this, we will add readiness probes to continuously monitor the connectivity between Autopilot and Prometheus. This will allow us to update the pod's readiness status (0/1) in real time, providing better visibility into potential issues and enabling proactive responses rather than waiting for an expansion failure

To address this, we are adding Readiness Probes instead of Liveness Probes. Liveness Probes would restart the pod if they fail, but in this case, a restart won't resolve the issue. Readiness Probes, on the other hand, will update the pod's readiness status (0/1) without triggering unnecessary restarts. This ensures continuous monitoring of the Autopilot-Prometheus connectivity and provides better visibility into potential issues, allowing for proactive responses.

A separate PR will be raised for Autopilot where /ready endpoint would serve the readiness prone hits made by k8s.

Which issue(s) this PR fixes (optional)

https://purestorage.atlassian.net/browse/PWX-38584

https://purestorage.atlassian.net/browse/PWX-38575

Testing

• Executed existing unit tests. No failures observed.

• Added unit tests for the code added.

• Deployed operator image with above change and autopilot:1.3.15. Verified that autopilot deployment does not has Readiness Probe.

## Autopilot container

Containers:
  autopilot:
    Container ID:  cri-o://218c6b7e430eeccff2337f7643b811de1b552f430cb08c9f4928a2deab5a23c3
    Image:         docker.io/portworx/autopilot:1.3.15
    Image ID:      docker.io/portworx/autopilot@sha256:50581ec6c42e30a5cfed7099bac05242bf943f044ee45e000d1b56333f570fb3
    Port:          <none>
    Host Port:     <none>
    Command:
      /autopilot
      --config=/etc/config/config.yaml
      --log-level=debug
    State:          Running
      Started:      Tue, 20 Aug 2024 16:30:55 +0000
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:  250m
    Requests:
      cpu:  100m
    Environment:
      PX_NAMESPACE:  kube-system
    Mounts:
      /etc/config from config-volume (rw)
      /etc/ssl/px-custom/1 from ca-cert-volume (ro)
      /var/cores from varcores (rw)
      /var/local/secrets from token-volume (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-25m2k (ro)

• Deployed operator image with above change and autopilot:1.3.16. Verified that autopilot deployment has Readiness Probe.

## Autopilot container

Containers:
  autopilot:
    Container ID:  cri-o://69122b8854be06976d094509c16e2f53bb798ffc86ab70c065f2bee66ab045db
    Image:         pure-artifactory.dev.purestorage.com/px-docker-dev-local/autopilot:1.3.16
    Image ID:      pure-artifactory.dev.purestorage.com/px-docker-dev-local/autopilot@sha256:eb312be27f80771abc3468bdafbadd00afc072860c5e01445dc7f208ce9d34c5
    Port:          <none>
    Host Port:     <none>
    Command:
      /autopilot
      --config=/etc/config/config.yaml
      --log-level=debug
    State:          Running
      Started:      Tue, 20 Aug 2024 16:53:21 +0000
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:  250m
    Requests:
      cpu:      100m
    Readiness:  http-get http://:9628/ready delay=10s timeout=5s period=10s #success=1 #failure=3
    Environment:
      PX_NAMESPACE:  kube-system
    Mounts:
      /etc/config from config-volume (rw)
      /etc/ssl/px-custom/1 from ca-cert-volume (ro)
      /var/cores from varcores (rw)
      /var/local/secrets from token-volume (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-4cxd9 (ro)

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 78.32%. Comparing base (30ed9ab) to head (df5da7b).
Report is 2 commits behind head on mdaigavane-feature-branch.

Additional details and impacted files

@@                      Coverage Diff                      @@
##           mdaigavane-feature-branch    #1654      +/-   ##
=============================================================
- Coverage                      79.79%   78.32%   -1.47%     
=============================================================
  Files                             61       61              
  Lines                          18831    18855      +24     
=============================================================
- Hits                           15026    14768     -258     
- Misses                          2715     3005     +290     
+ Partials                        1090     1082       -8     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[github-actions]

This PR is stale because it has been in review for 3 days with no activity.