-
Notifications
You must be signed in to change notification settings - Fork 337
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Adding ray as a distributor #272
Conversation
1d93049
to
b315c84
Compare
@rom1504 can you take a look and merge this :) |
Looks pretty good. Any way to add a test for this distributor ? |
Hi, is there any plan to merge this PR? |
@jelech I want to but I would prefer we add some testing as otherwise it'll eventually get broken over time |
Let me do it!
…________________________________
From: Romain Beaumont ***@***.***>
Sent: Saturday, July 15, 2023 2:41:14 PM
To: rom1504/img2dataset ***@***.***>
Cc: Vaishaal ***@***.***>; Author ***@***.***>
Subject: Re: [rom1504/img2dataset] Adding ray as a distributor (PR #272)
@jelech<https://github.com/jelech> I want to but I would prefer we add some testing as otherwise it'll eventually get broken over time
—
Reply to this email directly, view it on GitHub<#272 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAKMJEEKH4TQTX72XPRCWP3XQLP4VANCNFSM6AAAAAAT5LYA3Y>.
You are receiving this because you authored the thread.Message ID: ***@***.***>
|
I think we need this https://docs.ray.io/en/latest/cluster/vms/user-guides/launching-clusters/on-premises.html eg in gh action run
|
then adapt https://github.com/rom1504/img2dataset/blob/main/tests/test_main.py#L363 plus either automatically start a ray cluster if there is none in the main code or do it only in tests |
ok so I added tests I'll rework the doc here a little bit and then we're good for merging |
Nice thanks so much! |
thanks for the PR! |
For running large jobs on AWS I found a couple problems with the spark backend.
This PR just adds a ray distributor (with same interface as spark and multiprocessing distributors), and adds an example launch script, and a cluster_config.yaml file if people want to spin up their own AWS cluster.
Using this I was able to get over 200k/images second on a cluster of 100 m5.24xlarges consistently for 24 hours.