Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Job] Add a proper error message when ray[default] is not installed and job submission is used #41604

Open
rkooo567 opened this issue Dec 5, 2023 · 0 comments
Open

[Job] Add a proper error message when ray[default] is not installed and job submission is used #41604

rkooo567 opened this issue Dec 5, 2023 · 0 comments
Labels
bug Something that is supposed to be working; but isn't jobs P1 Issue that should be fixed within a few weeks

Comments

@rkooo567
Copy link
Contributor

rkooo567 commented Dec 5, 2023

What happened + What you expected to happen

I’ve got a ray job that I’m submitting with the command ray job submit --runtime-env-json='{"working_dir": "./"}' -- python predict.py write_embeddings=true  and it runs all the way to the end, runs the last line in the code ([logger.info](http://logger.info/)("Done")), and is even marked as SUCCEEDED on the ray dashboard, but the cli itself fails with this error:
Traceback (most recent call last):
  File "/Users/danturkel/.pyenv/versions/deeplike/bin/ray", line 8, in <module>
    sys.exit(main())
  File "/Users/danturkel/.pyenv/versions/3.10.8/envs/deeplike/lib/python3.10/site-packages/ray/scripts/scripts.py", line 2498, in main
    return cli()
  File "/Users/danturkel/.pyenv/versions/3.10.8/envs/deeplike/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/Users/danturkel/.pyenv/versions/3.10.8/envs/deeplike/lib/python3.10/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/Users/danturkel/.pyenv/versions/3.10.8/envs/deeplike/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/danturkel/.pyenv/versions/3.10.8/envs/deeplike/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/danturkel/.pyenv/versions/3.10.8/envs/deeplike/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/danturkel/.pyenv/versions/3.10.8/envs/deeplike/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/Users/danturkel/.pyenv/versions/3.10.8/envs/deeplike/lib/python3.10/site-packages/ray/dashboard/modules/job/cli_utils.py", line 54, in wrapper
    return func(*args, **kwargs)
  File "/Users/danturkel/.pyenv/versions/3.10.8/envs/deeplike/lib/python3.10/site-packages/ray/autoscaler/_private/cli_logger.py", line 856, in wrapper
    return f(*args, **kwargs)
  File "/Users/danturkel/.pyenv/versions/3.10.8/envs/deeplike/lib/python3.10/site-packages/ray/dashboard/modules/job/cli.py", line 306, in submit
    job_status = get_or_create_event_loop().run_until_complete(
  File "/Users/danturkel/.pyenv/versions/3.10.8/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/Users/danturkel/.pyenv/versions/3.10.8/envs/deeplike/lib/python3.10/site-packages/ray/dashboard/modules/job/cli.py", line 96, in _tail_logs
    return _log_job_status(client, job_id)
  File "/Users/danturkel/.pyenv/versions/3.10.8/envs/deeplike/lib/python3.10/site-packages/ray/dashboard/modules/job/cli.py", line 75, in _log_job_status
    info = client.get_job_info(job_id)
  File "/Users/danturkel/.pyenv/versions/3.10.8/envs/deeplike/lib/python3.10/site-packages/ray/dashboard/modules/job/sdk.py", line 346, in get_job_info
    return JobDetails(**r.json())
TypeError: 'NoneType' object is not callable

this has been resolved when pip install "ray[default]" is used.

Versions / Dependencies

master

Reproduction script

n/a

Issue Severity

None
@rkooo567 rkooo567 added bug Something that is supposed to be working; but isn't triage Needs triage (eg: priority, bug/not-bug, and owning component) jobs labels Dec 5, 2023
@architkulkarni architkulkarni added P1 Issue that should be fixed within a few weeks and removed triage Needs triage (eg: priority, bug/not-bug, and owning component) labels Dec 11, 2023
@architkulkarni architkulkarni removed their assignment Dec 11, 2023
orcahmlee added a commit to orcahmlee/ray that referenced this issue Dec 23, 2024
@orcahmlee
[Dashboard] Added a proper message when Ray jobs submission is used a…
5feee4f 
…nd the Pydantic is not installed (ray-project#41604)

Signed-off-by: Andrew <orcahmlee@gmail.com>
@orcahmlee orcahmlee mentioned this issue Dec 23, 2024
Merged
8 tasks
pcmoritz pushed a commit that referenced this issue Feb 25, 2025
@orcahmlee
[Dashboard] Added a proper message when Ray jobs submission is used a… (
eb45f89 
#49409)

<!-- Thank you for your contribution! Please review
https://github.com/ray-project/ray/blob/master/CONTRIBUTING.rst before
opening a pull request. -->

<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->

## Why are these changes needed?
I recently used the `ray job submit` CLI and got this confused error
message
```sh
TypeError: 'NoneType' object is not callable
```

<!-- Please give a short summary of the change and the problem this
solves. -->

As the #41604 mentioned, it need a proper error message instead of
raising `NoneType` error.

## Related issue number

<!-- For example: "Closes #1234" -->

Related to #41604

## Checks

- [x] I've signed off every commit(by using the -s flag, i.e., `git
commit -s`) in this PR.
- [x] I've run `scripts/format.sh` to lint the changes in this PR.
- [ ] I've included any doc changes needed for
https://docs.ray.io/en/master/.
- [ ] I've added any new APIs to the API Reference. For example, if I
added a
method in Tune, I've added it in `doc/source/tune/api/` under the
           corresponding `.rst` file.
- [ ] I've made sure the tests are passing. Note that there might be a
few flaky tests, see the recent failures at https://flakey-tests.ray.io/
- Testing Strategy
   - [ ] Unit tests
   - [ ] Release tests
   - [ ] This PR is not tested :(

Signed-off-by: Andrew <orcahmlee@gmail.com>
kevin85421 pushed a commit to kevin85421/ray that referenced this issue Feb 28, 2025
@orcahmlee @kevin85421
[Dashboard] Added a proper message when Ray jobs submission is used a… (
cafad9c 
ray-project#49409)

<!-- Thank you for your contribution! Please review
https://github.com/ray-project/ray/blob/master/CONTRIBUTING.rst before
opening a pull request. -->

<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->

## Why are these changes needed?
I recently used the `ray job submit` CLI and got this confused error
message
```sh
TypeError: 'NoneType' object is not callable
```

<!-- Please give a short summary of the change and the problem this
solves. -->

As the ray-project#41604 mentioned, it need a proper error message instead of
raising `NoneType` error.

## Related issue number

<!-- For example: "Closes ray-project#1234" -->

Related to ray-project#41604

## Checks

- [x] I've signed off every commit(by using the -s flag, i.e., `git
commit -s`) in this PR.
- [x] I've run `scripts/format.sh` to lint the changes in this PR.
- [ ] I've included any doc changes needed for
https://docs.ray.io/en/master/.
- [ ] I've added any new APIs to the API Reference. For example, if I
added a
method in Tune, I've added it in `doc/source/tune/api/` under the
           corresponding `.rst` file.
- [ ] I've made sure the tests are passing. Note that there might be a
few flaky tests, see the recent failures at https://flakey-tests.ray.io/
- Testing Strategy
   - [ ] Unit tests
   - [ ] Release tests
   - [ ] This PR is not tested :(

Signed-off-by: Andrew <orcahmlee@gmail.com>
Signed-off-by: kaihsun <kaihsun@anyscale.com>
Michaelhess17 pushed a commit to Michaelhess17/ray that referenced this issue Mar 3, 2025
@orcahmlee @Michaelhess17
[Dashboard] Added a proper message when Ray jobs submission is used a… (
841b36d 
ray-project#49409)

<!-- Thank you for your contribution! Please review
https://github.com/ray-project/ray/blob/master/CONTRIBUTING.rst before
opening a pull request. -->

<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->

## Why are these changes needed?
I recently used the `ray job submit` CLI and got this confused error
message
```sh
TypeError: 'NoneType' object is not callable
```

<!-- Please give a short summary of the change and the problem this
solves. -->

As the ray-project#41604 mentioned, it need a proper error message instead of
raising `NoneType` error.

## Related issue number

<!-- For example: "Closes ray-project#1234" -->

Related to ray-project#41604

## Checks

- [x] I've signed off every commit(by using the -s flag, i.e., `git
commit -s`) in this PR.
- [x] I've run `scripts/format.sh` to lint the changes in this PR.
- [ ] I've included any doc changes needed for
https://docs.ray.io/en/master/.
- [ ] I've added any new APIs to the API Reference. For example, if I
added a
method in Tune, I've added it in `doc/source/tune/api/` under the
           corresponding `.rst` file.
- [ ] I've made sure the tests are passing. Note that there might be a
few flaky tests, see the recent failures at https://flakey-tests.ray.io/
- Testing Strategy
   - [ ] Unit tests
   - [ ] Release tests
   - [ ] This PR is not tested :(

Signed-off-by: Andrew <orcahmlee@gmail.com>
xsuler pushed a commit to antgroup/ant-ray that referenced this issue Mar 4, 2025
@orcahmlee @xsuler
[Dashboard] Added a proper message when Ray jobs submission is used a… (
3198c5d 
ray-project#49409)

<!-- Thank you for your contribution! Please review
https://github.com/ray-project/ray/blob/master/CONTRIBUTING.rst before
opening a pull request. -->

<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->

## Why are these changes needed?
I recently used the `ray job submit` CLI and got this confused error
message
```sh
TypeError: 'NoneType' object is not callable
```

<!-- Please give a short summary of the change and the problem this
solves. -->

As the ray-project#41604 mentioned, it need a proper error message instead of
raising `NoneType` error.

## Related issue number

<!-- For example: "Closes ray-project#1234" -->

Related to ray-project#41604

## Checks

- [x] I've signed off every commit(by using the -s flag, i.e., `git
commit -s`) in this PR.
- [x] I've run `scripts/format.sh` to lint the changes in this PR.
- [ ] I've included any doc changes needed for
https://docs.ray.io/en/master/.
- [ ] I've added any new APIs to the API Reference. For example, if I
added a
method in Tune, I've added it in `doc/source/tune/api/` under the
           corresponding `.rst` file.
- [ ] I've made sure the tests are passing. Note that there might be a
few flaky tests, see the recent failures at https://flakey-tests.ray.io/
- Testing Strategy
   - [ ] Unit tests
   - [ ] Release tests
   - [ ] This PR is not tested :(

Signed-off-by: Andrew <orcahmlee@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something that is supposed to be working; but isn't jobs P1 Issue that should be fixed within a few weeks
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@architkulkarni @rkooo567