Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Fix] Avoid pickling entire LLMEngine for Ray workers #3207

Merged
merged 4 commits into from
Mar 6, 2024
Merged

[Fix] Avoid pickling entire LLMEngine for Ray workers #3207

merged 4 commits into from
Mar 6, 2024

Conversation

njhill
Copy link
Member

@njhill njhill commented Mar 5, 2024 •
edited
Loading

self references have been inadvertently introduced into the closure used to initialize the Ray workers, which is causing the whole LLMEngine object to be pickled with it.

This fixes it to reference the lora_config and kv_cache_dtype directly.

@njhill njhill mentioned this pull request Mar 5, 2024
Closed
Yard1
Yard1 approved these changes Mar 5, 2024
View reviewed changes
Copy link
Collaborator

@Yard1 Yard1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, can you add a short description?

@simon-mo
Copy link
Collaborator

simon-mo commented Mar 5, 2024

You can implement the __reduce__ method in LLMEngine and throw an error in there so this won't happen again in the future.

@njhill
Copy link
Member Author

njhill commented Mar 5, 2024

Good idea @simon-mo, now added.

Yard1
Yard1 reviewed Mar 5, 2024
View reviewed changes
@Yard1 Yard1 enabled auto-merge (squash) March 5, 2024 22:47
Yard1
Yard1 reviewed Mar 5, 2024
View reviewed changes
@Yard1 Yard1 merged commit 2efce05 into vllm-project:main Mar 6, 2024
22 checks passed
@njhill njhill deleted the dont-pickle-engine branch March 6, 2024 00:38
@grandiose-pizza grandiose-pizza mentioned this pull request Mar 6, 2024
Merged
dtransposed pushed a commit to afeldman-nm/vllm that referenced this pull request Mar 26, 2024
@njhill @Yard1
[Fix] Avoid pickling entire LLMEngine for Ray workers (vllm-project#3207
4c5cc5f 
)

Co-authored-by: Antoni Baum <antoni.baum@protonmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Yard1 Yard1 Yard1 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@njhill @simon-mo @Yard1