[k8s_cloud] Ray pod not created under current context namespace.

[aviweit]

Hi @romilbhardwaj ,

I encountered an issue where ray pods and services are not created under the namespace defined in the current context (by defining it via kubectl config set-context --current --namespace=<namespace name>).

This PR seems to fix the issue under my environment. Can you please review it?
Thanks.

Please find below what I did:

sky local up # wait for kind to create skypilot cluster
kubectl config set-context --current --namespace=test
kubectl create ns test
sky launch ../tasks/hi.yaml # very simple task that echoes

Ray pod and services are created under default namespace rather than test. The provision seems to hang since the secret is being created under test namespace.

Looking into it more, it seems that namespace of the current context is located under context key

e.g. in my kubeadm cluster

{'context': {'cluster': 'kubernetes', 'namespace': 'test', 'user': 'kubernetes-admin'}, 'name': 'kubernetes-admin@kubernetes'}

in my kind cluster

 {'context': {'cluster': 'kind-skypilot', 'namespace': 'test', 'user': 'kind-skypilot'}, 'name': 'kind-skypilot'}

[romilbhardwaj]

Thanks for catching this @aviweit! This requires one more change in get_head_ssh_port in backend_utils.py to work correctly (namespace is hardcoded to default there), but I'll go ahead and merge this and fix directly in k8s_cloud. Thanks again for finding this bug!

(base) ➜  ~ sky launch -c clus
I 07-25 16:54:54 optimizer.py:636] == Optimizer ==
I 07-25 16:54:54 optimizer.py:647] Target: minimizing cost
I 07-25 16:54:54 optimizer.py:659] Estimated cost: $0.0 / hour
I 07-25 16:54:54 optimizer.py:659]
I 07-25 16:54:54 optimizer.py:732] Considered resources (1 node):
I 07-25 16:54:54 optimizer.py:781] ---------------------------------------------------------------------------------------------------
I 07-25 16:54:54 optimizer.py:781]  CLOUD        INSTANCE          vCPUs   Mem(GB)   ACCELERATORS   REGION/ZONE   COST ($)   CHOSEN
I 07-25 16:54:54 optimizer.py:781] ---------------------------------------------------------------------------------------------------
I 07-25 16:54:54 optimizer.py:781]  Kubernetes   2CPU--2GB         2       2         -              kubernetes    0.00          ✔
I 07-25 16:54:54 optimizer.py:781]  AWS          m6i.2xlarge       8       32        -              us-east-1     0.38
I 07-25 16:54:54 optimizer.py:781]  Azure        Standard_D8s_v5   8       32        -              eastus        0.38
I 07-25 16:54:54 optimizer.py:781]  IBM          bx2-8x32          8       32        -              us-east       0.38
I 07-25 16:54:54 optimizer.py:781]  GCP          n2-standard-8     8       32        -              us-central1   0.39
I 07-25 16:54:54 optimizer.py:781]  Lambda       gpu_1x_a10        30      200       A10:1          us-east-1     0.60
I 07-25 16:54:54 optimizer.py:781] ---------------------------------------------------------------------------------------------------
I 07-25 16:54:54 optimizer.py:781]
Launching a new cluster 'clus'. Proceed? [Y/n]: Y
I 07-25 16:54:55 cloud_vm_ray_backend.py:3962] Creating a new cluster: "clus" [1x Kubernetes(2CPU--2GB)].
I 07-25 16:54:55 cloud_vm_ray_backend.py:3962] Tip: to reuse an existing cluster, specify --cluster (-c). Run `sky status` to see existing clusters.
I 07-25 16:54:55 cloud_vm_ray_backend.py:1409] To view detailed progress: tail -n100 -f /Users/romilb/sky_logs/sky-2023-07-25-16-54-53-205112/provision.log
I 07-25 16:54:55 cloud_vm_ray_backend.py:1762] Launching on Kubernetes
I 07-25 16:55:13 cloud_vm_ray_backend.py:1575] Successfully provisioned or found existing VM.
Clusters
NAME  LAUNCHED     RESOURCES                 STATUS  AUTOSTOP  COMMAND
clus  21 secs ago  1x Kubernetes(2CPU--2GB)  INIT    -         sky launch -c clus

Traceback (most recent call last):
  File "/Users/romilb/tools/anaconda3/bin/sky", line 8, in <module>
    sys.exit(cli())
  File "/Users/romilb/tools/anaconda3/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/Users/romilb/tools/anaconda3/lib/python3.9/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/Users/romilb/Romil/Berkeley/Research/sky-experiments/sky/utils/common_utils.py", line 239, in _record
    return f(*args, **kwargs)
  File "/Users/romilb/Romil/Berkeley/Research/sky-experiments/sky/cli.py", line 1112, in invoke
    return super().invoke(ctx)
  File "/Users/romilb/tools/anaconda3/lib/python3.9/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/romilb/tools/anaconda3/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/romilb/tools/anaconda3/lib/python3.9/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/Users/romilb/Romil/Berkeley/Research/sky-experiments/sky/utils/common_utils.py", line 260, in _record
    return f(*args, **kwargs)
  File "/Users/romilb/Romil/Berkeley/Research/sky-experiments/sky/cli.py", line 1380, in launch
    _launch_with_confirm(task,
  File "/Users/romilb/Romil/Berkeley/Research/sky-experiments/sky/cli.py", line 796, in _launch_with_confirm
    sky.launch(
  File "/Users/romilb/Romil/Berkeley/Research/sky-experiments/sky/utils/common_utils.py", line 260, in _record
    return f(*args, **kwargs)
  File "/Users/romilb/Romil/Berkeley/Research/sky-experiments/sky/utils/common_utils.py", line 260, in _record
    return f(*args, **kwargs)
  File "/Users/romilb/Romil/Berkeley/Research/sky-experiments/sky/execution.py", line 487, in launch
    _execute(
  File "/Users/romilb/Romil/Berkeley/Research/sky-experiments/sky/execution.py", line 324, in _execute
    handle = backend.provision(task,
  File "/Users/romilb/Romil/Berkeley/Research/sky-experiments/sky/utils/common_utils.py", line 260, in _record
    return f(*args, **kwargs)
  File "/Users/romilb/Romil/Berkeley/Research/sky-experiments/sky/utils/common_utils.py", line 239, in _record
    return f(*args, **kwargs)
  File "/Users/romilb/Romil/Berkeley/Research/sky-experiments/sky/backends/backend.py", line 56, in provision
    return self._provision(task, to_provision, dryrun, stream_logs,
  File "/Users/romilb/Romil/Berkeley/Research/sky-experiments/sky/backends/cloud_vm_ray_backend.py", line 2619, in _provision
    ssh_port_list = handle.external_ssh_ports(
  File "/Users/romilb/Romil/Berkeley/Research/sky-experiments/sky/backends/cloud_vm_ray_backend.py", line 2310, in external_ssh_ports
    self._update_stable_ssh_ports(max_attempts=max_attempts)
  File "/Users/romilb/Romil/Berkeley/Research/sky-experiments/sky/backends/cloud_vm_ray_backend.py", line 2229, in _update_stable_ssh_ports
    head_port = backend_utils.get_head_ssh_port(
  File "/Users/romilb/Romil/Berkeley/Research/sky-experiments/sky/utils/common_utils.py", line 260, in _record
    return f(*args, **kwargs)
  File "/Users/romilb/Romil/Berkeley/Research/sky-experiments/sky/backends/backend_utils.py", line 1622, in get_head_ssh_port
    head_ssh_port = clouds.Kubernetes.get_port(svc_name, 'default')
  File "/Users/romilb/Romil/Berkeley/Research/sky-experiments/sky/clouds/kubernetes.py", line 217, in get_port
    return kubernetes_utils.get_port(svc_name, namespace)
  File "/Users/romilb/Romil/Berkeley/Research/sky-experiments/sky/skylet/providers/kubernetes/utils.py", line 15, in get_port
    head_service = kubernetes.core_api().read_namespaced_service(
  File "/Users/romilb/tools/anaconda3/lib/python3.9/site-packages/kubernetes/client/api/core_v1_api.py", line 24931, in read_namespaced_service
    return self.read_namespaced_service_with_http_info(name, namespace, **kwargs)  # noqa: E501
  File "/Users/romilb/tools/anaconda3/lib/python3.9/site-packages/kubernetes/client/api/core_v1_api.py", line 25018, in read_namespaced_service_with_http_info
    return self.api_client.call_api(
  File "/Users/romilb/tools/anaconda3/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 348, in call_api
    return self.__call_api(resource_path, method,
  File "/Users/romilb/tools/anaconda3/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 180, in __call_api
    response_data = self.request(
  File "/Users/romilb/tools/anaconda3/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 373, in request
    return self.rest_client.GET(url,
  File "/Users/romilb/tools/anaconda3/lib/python3.9/site-packages/kubernetes/client/rest.py", line 241, in GET
    return self.request("GET", url,
  File "/Users/romilb/tools/anaconda3/lib/python3.9/site-packages/kubernetes/client/rest.py", line 235, in request
    raise ApiException(http_resp=r)
kubernetes.client.exceptions.ApiException: (404)
Reason: Not Found
HTTP response headers: HTTPHeaderDict({'Audit-Id': '4dde30ba-2773-442c-a2ad-c8b76836a3d1', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': 'f7320e7d-cba7-43fd-a67f-d02390250a65', 'X-Kubernetes-Pf-Prioritylevel-Uid': '3b49a09f-6452-4859-99bc-ffe73df162b2', 'Date': 'Tue, 25 Jul 2023 23:55:15 GMT', 'Content-Length': '210'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"services \"clus-ray-head-ssh\" not found","reason":"NotFound","details":{"name":"clus-ray-head-ssh","kind":"services"},"code":404}