Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Docker] docker mode fails when nvidia-smi is not installed #418

Closed
romilbhardwaj opened this issue Feb 23, 2022 · 1 comment · Fixed by #419
Closed

[Docker] docker mode fails when nvidia-smi is not installed #418

romilbhardwaj opened this issue Feb 23, 2022 · 1 comment · Fixed by #419
Assignees

Comments

@romilbhardwaj
Copy link
Collaborator

romilbhardwaj commented Feb 23, 2022

After #411, LocalDockerBackend is broken for users who don't have nvidia-smi installed. Report from @gmittal:

sky ❯ sky launch examples/minimal.yaml --docker
Task from YAML spec: examples/minimal.yaml
Traceback (most recent call last):
  File "/Users/gautam/miniconda3/envs/sky/bin/sky", line 33, in <module>
    sys.exit(load_entry_point('sky', 'console_scripts', 'sky')())
  File "/Users/gautam/miniconda3/envs/sky/lib/python3.8/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/Users/gautam/miniconda3/envs/sky/lib/python3.8/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/Users/gautam/miniconda3/envs/sky/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/gautam/miniconda3/envs/sky/lib/python3.8/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/gautam/miniconda3/envs/sky/lib/python3.8/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/Users/gautam/Desktop/workbench/sky/sky/cli.py", line 521, in launch
    backend = backends.LocalDockerBackend()
  File "/Users/gautam/Desktop/workbench/sky/sky/backends/local_docker_backend.py", line 108, in __init__
    self._use_gpu = backend_utils.check_local_gpus() if use_gpu == 'auto' \
  File "/Users/gautam/Desktop/workbench/sky/sky/backends/backend_utils.py", line 752, in check_local_gpus
    execution_check = subprocess.run(['nvidia-smi'],
  File "/Users/gautam/miniconda3/envs/sky/lib/python3.8/subprocess.py", line 493, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/Users/gautam/miniconda3/envs/sky/lib/python3.8/subprocess.py", line 858, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/Users/gautam/miniconda3/envs/sky/lib/python3.8/subprocess.py", line 1704, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'nvidia-smi'

this line is likely culprit:
https://github.com/sky-proj/sky/blob/d19a6fd2a5b7f7da3d1b3bfa2c2fb40c9d90b27f/sky/backends/backend_utils.py#L751

It should just check:
if is_installed:

@romilbhardwaj romilbhardwaj changed the title nvidia-smi fails on docker mode fails when nvidia-smi is not installed Feb 23, 2022
@romilbhardwaj romilbhardwaj changed the title docker mode fails when nvidia-smi is not installed [Docker] docker mode fails when nvidia-smi is not installed Feb 23, 2022
@WoosukKwon
Copy link
Collaborator

WoosukKwon commented Feb 23, 2022

Oh I'm really sorry. Thanks for reporting this problem @gmittal @romilbhardwaj.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants