Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Misc] allow user to specify where to write gpu_p2p_access_cache through VLLM_CACHE_DIR env var #4491

Closed
wants to merge 3 commits into from
Closed

[Misc] allow user to specify where to write gpu_p2p_access_cache through VLLM_CACHE_DIR env var #4491

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
39e86fa
wip
sfc-gh-zhwang Apr 30, 2024
8acbc93
remove debug
sfc-gh-zhwang Apr 30, 2024
6c3f0dc
yapf
sfc-gh-zhwang Apr 30, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions vllm/distributed/utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -105,8 +105,10 @@ def gpu_p2p_access_check(i: int, j: int) -> bool:
cuda_visible_devices = os.environ.get("CUDA_VISIBLE_DEVICES", None)
if cuda_visible_devices is None:
cuda_visible_devices = ",".join(str(i) for i in range(num_dev))
path = os.path.expanduser(
f"~/.config/vllm/gpu_p2p_access_cache_for_{cuda_visible_devices}.json")
cache_dir = os.environ.get(
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

now i see it, looks like the whole logic is not even needed, we can just populate _gpu_p2p_access_cache?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you mean by the whole logic is not even needed ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I meant, we don't need to dump a json to disk, instead we just populate _gpu_p2p_access_cache if it's None, otherwise, we just read from _gpu_p2p_access_cache. Am i missing anything?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

save to disk is useful to save memory, unless you want the first rank of the worker to have less gpu memory: #4021 .

"VLLM_CACHE_DIR", "").strip() or os.path.expanduser("~/.config/vllm/")
path = os.path.join(
cache_dir, f"gpu_p2p_access_cache_for_{cuda_visible_devices}.json")
os.makedirs(os.path.dirname(path), exist_ok=True)
if (not is_distributed or get_local_rank() == 0) \
and (not os.path.exists(path)):
Expand Down
Loading