Change Ray's default port to avoid conflicts with Redis #1790

infwinston · 2023-03-16T21:43:45Z

Fixes #1789. I looked up common ports used by well-known services here
https://en.wikipedia.org/wiki/List_of_TCP_and_UDP_port_numbers
and 6380 seems safe.

This PR also enables Ray applications on SkyPilot by making our Ray cluster invisible by changing the default temp_dir to /tmp/ray_skypilot. (related #1305, #671)

Tested (run the relevant ones):

All smoke tests: pytest tests/test_smoke.py
Backward compatibility tests: bash tests/backward_comaptibility_tests.sh

suquark · 2023-03-18T00:24:57Z

we should use ray_skypilot instead of ray_patch as the temp dir name, in case this becomes our long-term solution

concretevitamin

Awesome! Quick look. ray start -h shows some other ports to check. Is it ok to share them with an app's Ray?

  --object-manager-port INTEGER   the port to use for starting the object
                                  manager
  --node-manager-port INTEGER     the port to use for starting the node
                                  manager
  --gcs-server-port INTEGER       Port number for the GCS server.
  --min-worker-port INTEGER       the lowest port number that workers will
                                  bind on. If not set, random ports will be
                                  chosen.
  --max-worker-port INTEGER       the highest port number that workers will
                                  bind on. If set, '--min-worker-port' must
                                  also be set.
  --worker-port-list TEXT         a comma-separated list of open ports for
                                  workers to bind on. Overrides '--min-worker-
                                  port' and '--max-worker-port'.
  --ray-client-server-port INTEGER
                                  the port number the ray client server binds
                                  on, default to 10001.
...
  --dashboard-port INTEGER        the port to bind the dashboard server to--
                                  defaults to 8265
  --dashboard-agent-listen-port INTEGER
                                  the port for dashboard agents to listen for
                                  http on.
  --dashboard-agent-grpc-port INTEGER
                                  the port for dashboard agents to listen for
                                  grpc on.
  --metrics-export-port INTEGER   the port to use to expose Ray metrics
                                  through a Prometheus endpoint.

May want to run

https://github.com/skypilot-org/skypilot/tree/master/examples/ray_tune_examples
AutoGluon
back compat

concretevitamin · 2023-03-30T18:38:11Z

sky/templates/azure-ray.yml.j2

@@ -143,12 +143,12 @@ head_start_ray_commands:
  # Start skylet daemon. (Should not place it in the head_setup_commands, otherwise it will run before skypilot is installed.)
  # NOTE: --disable-usage-stats in `ray start` saves 10 seconds of idle wait.
  - ((ps aux | grep -v nohup | grep -v grep | grep -q -- "python3 -m sky.skylet.skylet") || nohup python3 -m sky.skylet.skylet >> ~/.sky/skylet.log 2>&1 &);
-    ray stop; RAY_SCHEDULER_EVENTS=0 ray start --disable-usage-stats --head --port=6379 --object-manager-port=8076 --autoscaling-config=~/ray_bootstrap_config.yaml {{"--resources='%s'" % custom_resources if custom_resources}} || exit 1;
+    ray stop; RAY_SCHEDULER_EVENTS=0 ray start --disable-usage-stats --head --port={{ray_port}} --object-manager-port=8076 --autoscaling-config=~/ray_bootstrap_config.yaml {{"--resources='%s'" % custom_resources if custom_resources}} --temp-dir {{ray_temp_dir}} || exit 1;


Is it ok that our Ray continues to use --object-manager-port=8076?

infwinston · 2023-04-02T21:39:10Z

Is it ok that our Ray continues to use --object-manager-port=8076?

To my understanding, from their doc,object-manager-port by default will be randomly decided in runtime (in which user's ray will use a port different from 8076).
so it'd not cause issue. But @Michaelvll do you know why we use a specific port 8076 in the first place?

I'll upload an AutoGluon example I run before.

Back compat should be fine if we keep our dashboard running on the same port 8265, which is used to submit our Ray job. However, if user's Ray app requires dashboard (default port: 8265), it'll still be conflicts. We may need to decide whether to change our dashboard port. each side comes with pros/cons.

Btw, I'm also blocked by this issue in vicuna project in which I use ray to distribute work to multiple gpus. Now my sky yaml fails to run my app. We may want to merge this PR soon.

Michaelvll · 2023-04-11T06:30:31Z

For the backward compatibility of the 8265 port, in our new implementation, we can start the ray cluster with dashboard port 8266 and note down the port in a file. When we are submitting the task or querying the ray cluster for the job status, we can look at the file first and decide which port to use for them:
In ray yaml:
ray stop; RAY_SCHEDULER_EVENTS=0 ray start --disable-usage-stats --head --port={{ray_port}} --object-manager-port=8076 --dashboard-port 8266 --autoscaling-config=~/ray_bootstrap_config.yaml {{"--resources='%s'" % custom_resources if custom_resources}} --temp-dir {{ray_temp_dir}} && echo '8266' > ~/.sky/ray-dashboard-port || exit 1;`

For job submission here

skypilot/sky/backends/cloud_vm_ray_backend.py

Lines 2611 to 2615 in 15f01d6

    
           job_submit_cmd = ( 
        
               f'{cd} && mkdir -p {remote_log_dir} && ray job submit ' 
        
               f'--address=http://127.0.0.1:8265 --submission-id {ray_job_id} ' 
        
               '--no-wait ' 
        
               f'"{executable} -u {script_path} > {remote_log_path} 2>&1"')

We can do something like the following:

job_submit_cmd = (
                f'RAY_DASHBOARD_PORT=$(cat ~/.sky/ray_dashboard_port 2> /dev/null); RAY_DASHBOARD_PORT=${RAY_DASHBOARD_PORT:-8265}; {cd} && mkdir -p {remote_log_dir} && ray job submit '
                f'--address=http://127.0.0.1:$RAY_DASHBOARD_PORT --submission-id {ray_job_id} '
                '--no-wait '
                f'"{executable} -u {script_path} > {remote_log_path} 2>&1"')

iojw · 2023-04-13T18:05:44Z

Tested and confirmed that the following backwards compatibility cases work:

exec on existing up cluster - as discussed, this depends on the job manager port, so changing the GCS port does not affect the submission of jobs
start on existing stopped cluster - this re-provisions the cluster using the new GCS port, so there is no issue here

Michaelvll · 2023-05-09T19:09:49Z

A kind bump for this PR @infwinston. Any updates?

infwinston · 2023-05-09T19:12:34Z

Will get back to office and look into this tomorrow.

infwinston · 2023-05-13T00:13:31Z

I've updated this PR to ensure backward compatibility! and tested it. seems to work fine.

Michaelvll

Thanks for the fix @infwinston! This would be very useful for users with their own ray cluster. Left several comments.

examples/autogluon.yaml

Michaelvll · 2023-05-23T22:59:18Z

sky/backends/backend_utils.py

+    port_dict_str = f'{{"ray_port":{ray_port}, "ray_dashboard_port":{ray_dashboard_port}}}'
+    dump_port_command = f'python -c \'import json, os; json.dump({port_dict_str}, open(os.path.expanduser("{constants.SKY_REMOTE_RAY_PORT_FILE}"), "w"))\''


nit: Since we are already using python, should we just do the following:

port_json = json.dumps({'ray_port': ray_port, 'ray_dashboard_port': ray_dashboard_port}) dump_port_command = f'echo {port_json!r} > {constants.SKY_REMOTE_RAY_PORT_FILE}'

Ah I tried this before. The issue is json.dumps will add a space between ":" and the value. which will cause the YAML parser to fail... we need "{'ray_port':6380}" not "{'ray_port': 6380}"..

I've added some comments to explain it.

Ahh, got it. I suppose if we add a | after the - in the head_start_ray_commands, the problem should be fixed. : )

Michaelvll · 2023-05-23T23:03:18Z

sky/backends/cloud_vm_ray_backend.py

+                'RAY_DASHBOARD_PORT=$(jq ".ray_dashboard_port // empty" '
+                f'{ray_port_file} 2> /dev/null); '


Does jq installed by default on all the images we used for all the clouds? If not, we may want to use python instead? Something like the following:

'RAY_DASHBOARD_POART=$(python -c "import json; print(json.load(open(\'{ray_port_file}\')).get(\'ray_dashboard_port\', 8265);" 2> /dev/null || echo 8265)'

or just

'RAY_DASHBOARD_POART=$(python -c "from sky.skylet import job_lib; print(job_lib.get_job_submission_port()) 2> /dev/null || echo 8265)'

Good point! updated.

sky/skylet/constants.py

Michaelvll · 2023-05-23T23:57:51Z

sky/backends/cloud_vm_ray_backend.py

+                address={ray_address!r},
+                namespace='__sky__{job_id}__',
+                log_to_driver=True,
+                _temp_dir={constants.SKY_REMOTE_RAY_TEMPDIR!r}


Will this work if we are submitting the job through sky exec to an existing cluster with the ray cluster running under the default _temp_dir? Should we check the existence of constants.SKY_REMOTE_TEMPDIR on the remote VM before we ray.init?

kwargs = {} if os.path.exists(constants.SKY_REMOTE_RAY_TEMPDIR): kwargs['_temp_dir'] = constants.SKY_REMOTE_RAY_TEMPDIR ray.init(..., **kwargs)

Good point. Added!

infwinston · 2023-05-27T19:57:07Z

I've merged the master and addressed the comment.

Tested:

sky launch examples/autogluon.yaml works.
Backward compatibility: I switched to master and launched a VM. and switch back to this branch. sky exec still works.
Smoke tests all passed

@Michaelvll could you review if there's anything left? thanks!

Michaelvll

Thanks for the fix @infwinston! LGTM. It should be good to go if the tests are passed.

sky/backends/cloud_vm_ray_backend.py

Co-authored-by: Zhanghao Wu <zhanghao.wu@outlook.com>

infwinston · 2023-05-28T00:13:51Z

Thanks a lot for the reviews @Michaelvll! Now all the smoke tests passed. Merging this.

infwinston added 4 commits March 16, 2023 21:37

init

0057915

comment

2b36e39

update

7f29a70

make ray invisible

490b199

update

99125d8

concretevitamin added this to the 0.3 milestone Mar 22, 2023

concretevitamin reviewed Mar 30, 2023

View reviewed changes

infwinston added 3 commits April 4, 2023 20:50

add autogluon example

5cea121

fix

4e5ae44

fix

1b462ff

Michaelvll self-requested a review April 8, 2023 07:39

Michaelvll linked an issue May 9, 2023 that may be closed by this pull request

User Ray and Sky Ray version conflicts #671

Closed

Michaelvll linked an issue May 12, 2023 that may be closed by this pull request

Multi-node ray.tune can only be run in default python #310

Closed

Michaelvll mentioned this pull request May 12, 2023

Multi-node ray.tune can only be run in default python #310

Closed

infwinston added 7 commits May 12, 2023 23:31

fix backward compatibility

be89847

fix

0854c39

add other clouds

98c649e

format

ca1e583

fix

72f8990

Merge branch 'master' into change-ray-port

f2ed96a

add ibm

71f5fc7

Michaelvll reviewed May 24, 2023

View reviewed changes

merge

c2663f0

infwinston added 2 commits May 27, 2023 19:34

comments

b29b98d

format

d4a45c6

Michaelvll approved these changes May 27, 2023

View reviewed changes

sky/backends/cloud_vm_ray_backend.py Show resolved Hide resolved

Update sky/backends/cloud_vm_ray_backend.py

b507f34

Co-authored-by: Zhanghao Wu <zhanghao.wu@outlook.com>

infwinston merged commit 4bd9d44 into master May 28, 2023

infwinston deleted the change-ray-port branch May 28, 2023 00:14

This was referenced May 29, 2023

[dependencies] user's ray program missing dependencies #1305

Closed

[Core] User ray cluster causes SkyPilot cluster in INIT state #2020

Merged

Michaelvll mentioned this pull request Nov 20, 2023

[Core] Install SkyPilot runtime in separate env #2801

Closed

19 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Change Ray's default port to avoid conflicts with Redis #1790

Change Ray's default port to avoid conflicts with Redis #1790

infwinston commented Mar 16, 2023 •

edited

Loading

suquark commented Mar 18, 2023

concretevitamin left a comment

concretevitamin Mar 30, 2023

infwinston commented Apr 2, 2023 •

edited

Loading

Michaelvll commented Apr 11, 2023 •

edited

Loading

iojw commented Apr 13, 2023

Michaelvll commented May 9, 2023 •

edited

Loading

infwinston commented May 9, 2023

infwinston commented May 13, 2023 •

edited

Loading

Michaelvll left a comment

Michaelvll May 23, 2023

infwinston May 27, 2023

Michaelvll May 27, 2023 •

edited

Loading

Michaelvll May 23, 2023

infwinston May 27, 2023

Michaelvll May 23, 2023

infwinston May 27, 2023

infwinston commented May 27, 2023 •

edited

Loading

Michaelvll left a comment

infwinston commented May 28, 2023

		port_dict_str = f'{{"ray_port":{ray_port}, "ray_dashboard_port":{ray_dashboard_port}}}'
		dump_port_command = f'python -c \'import json, os; json.dump({port_dict_str}, open(os.path.expanduser("{constants.SKY_REMOTE_RAY_PORT_FILE}"), "w"))\''

		'RAY_DASHBOARD_PORT=$(jq ".ray_dashboard_port // empty" '
		f'{ray_port_file} 2> /dev/null); '

Change Ray's default port to avoid conflicts with Redis #1790

Change Ray's default port to avoid conflicts with Redis #1790

Conversation

infwinston commented Mar 16, 2023 • edited Loading

suquark commented Mar 18, 2023

concretevitamin left a comment

Choose a reason for hiding this comment

concretevitamin Mar 30, 2023

Choose a reason for hiding this comment

infwinston commented Apr 2, 2023 • edited Loading

Michaelvll commented Apr 11, 2023 • edited Loading

iojw commented Apr 13, 2023

Michaelvll commented May 9, 2023 • edited Loading

infwinston commented May 9, 2023

infwinston commented May 13, 2023 • edited Loading

Michaelvll left a comment

Choose a reason for hiding this comment

Michaelvll May 23, 2023

Choose a reason for hiding this comment

infwinston May 27, 2023

Choose a reason for hiding this comment

Michaelvll May 27, 2023 • edited Loading

Choose a reason for hiding this comment

Michaelvll May 23, 2023

Choose a reason for hiding this comment

infwinston May 27, 2023

Choose a reason for hiding this comment

Michaelvll May 23, 2023

Choose a reason for hiding this comment

infwinston May 27, 2023

Choose a reason for hiding this comment

infwinston commented May 27, 2023 • edited Loading

Michaelvll left a comment

Choose a reason for hiding this comment

infwinston commented May 28, 2023

infwinston commented Mar 16, 2023 •

edited

Loading

infwinston commented Apr 2, 2023 •

edited

Loading

Michaelvll commented Apr 11, 2023 •

edited

Loading

Michaelvll commented May 9, 2023 •

edited

Loading

infwinston commented May 13, 2023 •

edited

Loading

Michaelvll May 27, 2023 •

edited

Loading

infwinston commented May 27, 2023 •

edited

Loading