Ensure port forwards do not select terminating pods and are ready before use

[jbohanon]

Description

This PR aims to fix flakiness in testing code which relies on port-forwards.

Code changes

• Check that pods are not in terminating status before attempting to port-forward.
  • The cli portforwarder does this by using a util to make sure no pods show Terminating from a kubectl get
  • The api portforwarder does this by making sure the pod resource doesn't have a deletionTimestamp

Context

Port forwards in Gloo have been historically flaky. There are two main root causes for this that were uncovered in this investigation:

• The function that starts the port-forward could return nil (no error) before the port-forward was fully initialized. This would lead to errors when the calling code tried to dial on the forwarded address immediately after receiving the nil return.
• The pod selection code allowed for pods in Terminating status to be selected as the target for the port-forward. This would lead to errors when the selected Terminating pod was finally killed and the port-forward failed.

Testing steps

TODO

Checklist:

• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have added tests that prove my fix is effective or that my feature works

[solo-changelog-bot]

Issues linked to changelog:
#9353

[github-actions]

Visit the preview URL for this PR (updated for commit 86c477b):

https://gloo-edge--pr9628-jbohanon-port-forwar-nw9o18rd.web.app

_{(expires Wed, 07 Aug 2024 15:13:11 GMT)}

_{🔥 via Firebase Hosting GitHub Action 🌎}

_{Sign: 77c2b86e287749579b7ff9cadb81e099042ef677}

[jbohanon]

Closing as the proposed changes have caused timeouts in kubernetes e2e tests, and this has been deprioritized due to the relative pain of the flake being low.

The work to get this done is still being tracked by #9353