When admission webhook and gateway discovery is enabled, actions requiring admission webhook stucks when gateway pod deleted

[randmonkey]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

When validation admission webhook is turned on, while kic and kong gateway are deployed in different deployments/pods, actions requiring admission webhooks will be stuck when kong gateway pod is deleted and re-created and got a different IP.
I think it is caused by the way the admission webhook server access the kong admin API. It uses a fixed address on setup phase, which could change when KIC and kong are in different pods.

Expected Behavior

When kong pod is deleted and re-created (at least one kong gateway pod available), webhook could still work.

Steps To Reproduce

1. Deploy multiple GW, with admission webhook enabled:
   add the env CONTROLLER_ADMISSION_WEBHOOK_LISTEN="0.0.0.0:8080" to `all-in-one-dbless-multiple-gw.yaml`
2. create service and admission webhook:
apiVersion: v1
kind: Service
metadata:
  name: kong-validation-webhook
  namespace: kong
spec:
  ports:
  - name: webhook
    port: 443
    protocol: TCP
    targetPort: 8080
  selector:
    app: ingress-kong
and validation admission webhook:
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: "validate.konghq.com"
webhooks:
- name: "plugin.konghq.com"
  rules:
  - apiGroups:   ["configuration.konghq.com"]
    apiVersions: ["v1"]
    operations:  ["CREATE"]
    resources:   ["kongplugins"]
    scope:       "Namespaced"
  clientConfig:
    service:
      namespace: "kong"
      name: "kong-validation-webhook"
   ...
To enable validation webhook.
3. delete the initial kong pods
4. create a kongPlugin, and k8s api will return error from validation webhook:
rror from server (InternalError): error when creating "kong-plugin-good.yaml": Internal error occurred: failed calling webhook "plugin.konghq.com": failed to call webhook: Post "https://kong-validation-webhook.kong.svc:443/?timeout=5s": context deadline exceeded

Kong Ingress Controller version

current main branch

Kubernetes version

No response

Anything else?

No response