Skip to content
This repository was archived by the owner on Oct 11, 2024. It is now read-only.

Commit 046eb08

Browse files
[ README ] Update README.md (#323)
1 parent d14c46e commit 046eb08

File tree

1 file changed

+15
-8
lines changed

1 file changed

+15
-8
lines changed

README.md

+15-8
Original file line numberDiff line numberDiff line change
@@ -1,17 +1,13 @@
1-
<h1 style="display: flex; align-items: center;" >
2-
<img width="100" height="100" alt="tool icon" src="https://neuralmagic.com/wp-content/uploads/2024/04/icon_nm_vllm-002-copy.svg" />
3-
<span>&nbsp;&nbsp;nm-vllm</span>
4-
</h1>
1+
# nm-vllm
52

63
## Overview
74

8-
[vLLM](https://github.com/vllm-project/vllm) is a fast and easy-to-use library for LLM inference that Neural Magic regularly contributes to.
9-
10-
`nm-vllm` is our supported enterprise distribution of vLLM.
5+
`nm-vllm` is our supported enterprise distribution of [vLLM](https://github.com/vllm-project/vllm).
116

127
## Installation
138

14-
The [nm-vllm PyPi package](https://pypi.neuralmagic.com/simple/nm-vllm/index.html) includes pre-compiled binaries for CUDA (version 12.1) kernels, streamlining the setup process. For other PyTorch or CUDA versions, please compile the package from source.
9+
### PyPI
10+
The [nm-vllm PyPi package](https://pypi.neuralmagic.com/simple/nm-vllm/index.html) includes pre-compiled binaries for CUDA (version 12.1) kernels. For other PyTorch or CUDA versions, please compile the package from source.
1511

1612
Install it using pip:
1713
```bash
@@ -30,6 +26,17 @@ cd nm-vllm
3026
pip install -e .[sparse] --extra-index-url https://pypi.neuralmagic.com/simple
3127
```
3228

29+
### Docker
30+
31+
The [`nm-vllm` container registry](https://github.com/neuralmagic/nm-vllm/pkgs/container/nm-vllm-openai) includes premade docker images.
32+
33+
Launch the OpenAI-compatible server with:
34+
35+
```bash
36+
MODEL_ID=Qwen/Qwen2-0.5B-Instruct
37+
docker run --gpus all --shm-size 2g ghcr.io/neuralmagic/nm-vllm-openai:latest --model $MODEL_ID
38+
```
39+
3340
## Models
3441

3542
Neural Magic maintains a variety of optimized models on our Hugging Face organization profiles:

0 commit comments

Comments
 (0)