Stabilize port forwarding

[priyawadhwa]

We've seen many issues around port-forwarding with the last few releases:

Port forwarding doesn't work anymore since update 0.33.0 -> 0.34.0 (#2523)
Port forwarding not work when pods recreated (#2408)

This issue will detail a plan to fix known issues and stabilize port forwarding.

1. (In progress - Add pods stability for health check. #2374) Implement pod health check. This way, we will wait for pods to be running before attempting port forwarding, which should prevent the "Pod Status = Pending" error in kubectl port-forward I saw when reproducing this comment

2. (Depends on Add pods stability for health check. #2374) Make sure pod is running before sending a port forward event for the IDEs even on retries

3. Add termination events for port forwards (Add termination events for port forwarding #2584)

4. (Depends on Point 2) Clarify the port forwarding log message to indicate we are attempting port forwarding (Clarify port forwarding log message #2585)

5. (Done - Add port forwarding unit & integration tests #2596) Add integration tests for Cleaner kubectl port-forward retry logic #2593

6. (Done with Select resources by UUID label #2609) Filter services by run ID instead of by K8sManagedByLabel, so that extra services running in cluster aren't port forwarded

7. (Done with Don't look up services in all namespaces. #2651) Make sure portforwarding doesn't require access to all namespaces

8. Consider a redesign if the above steps don't stabilize port forwarding to our satisfaction.

[balopat]

we should address #2599 as well here