[ Frontend ] Multiprocessing for OpenAI Server with zeromq
#6883
Commits on Jul 25, 2024
-
Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>
-
Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>
-
Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>
-
-
-
-
Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>
Commits on Jul 26, 2024
-
-
-
-
-
Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>
-
-
-
-
-
-
Commits on Jul 28, 2024
Commits on Jul 29, 2024
-
-
-
-
-
-
-
-
-
-
-
-
🥅 handle shutdown and request errors
Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>
-
🎨 fmt and clean up shutdown handler
Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>
-
Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>
-
Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>
-
Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>
-
Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>
-
Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>
Commits on Jul 30, 2024
-
@robertgshaw2-neuralmagic This adds the `--disable-frontend-multiprocessing` flag and should also correctly pick up embeddings models to disable the multiprocessing here. (Also some unrelated formatting changes) The backend stuff is wrapped up in a context manager that handles the process startup and shutdown at exit as well, so that we don't have to muck around much in the existing server lifecycle code --------- Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>
Commits on Jul 31, 2024
-
Features / Cleanup for MP Frontend (#387)
SUMMARY: * refactor to use single socket * cleanup comments / logging * add `do_log_stats` * add `abort`
-
Use random port for backend (#390)
Picks an open port to use and boots both the client and server with it --------- Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>
-
-
With all the extra fun refactors Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>
-
SUMMARY: * add endpoints to request `ModelConfig`, `SchedulerConfig`, `LoRAConfig`, `ParallelConfig` * factor out tokenizer group creation function to be a utility function * create tokenizer_group on client side
-
Ensures no sockets are leaked on the client-side Also postpones the server shutdown await so that the backend can shutdown concurrently, and all connections can be cleaned up at the same time. This prevents hangs where the frontend blocks on remaining connections but the backend has not yet initiated shutdown --------- Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>
-
SUMMARY: * fix issue with logit bias loading
-
🐛 messed up the revert in the merge commit :(
Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>
Commits on Aug 1, 2024
-
-
-
-
-
-
-
✅ add test for multiprocessing flag (#399)
Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>
-
(plus rounding out the protocol with an error on `.encode`) --------- Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>
-
-
-
-
-
-
Update vllm/entrypoints/openai/rpc/server.py
Co-authored-by: Simon Mo <simon.mo@hey.com>
-
Commits on Aug 2, 2024
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.