[CI/Build] build on empty device for better dev experience #4773

tomeras91 · 2024-05-12T14:46:18Z

This PR enables build of a platform-agnostic wheel which is installable also on macos. The idea is to improve the dev-experience for creating projects that import and use vLLM.
Important: This wheel does not enable running of vllm on mac, but does allow to import it.

The PR doesn't entirely fix issues #212, #695, #1397, #1921, but it's a step forward.

…e fixed

tomeras91

Curious to hear your thoughts.. currently the PR just enables building of the platform agnostic wheel. What do you think about adding this build to publish.yml so it will be build on every release?

tomeras91 · 2024-05-12T14:47:00Z

requirements-cpu.txt

@@ -2,5 +2,4 @@
 -r requirements-common.txt

 # Dependencies for x86_64 CPUs
-torch == 2.3.0+cpu
-triton >= 2.2.0  # FIXME(woosuk): This is a hack to avoid import error.


Dealt with triton import errors in code

simon-mo · 2024-05-23T21:01:31Z

Interestingly Triton recently supported macOS with Apple Silicon: triton-lang/triton#3443

tomeras91 · 2024-05-23T21:27:45Z

Interestingly Triton recently supported macOS with Apple Silicon: triton-lang/triton#3443

Yep. But AFAIU they still release wheels only for Linux so it can't be pip installed directly from pypi. I guess building from source is the only option to get triton installed on mac.
https://pypi.org/project/triton/2.3.0/#files

hugolytics · 2024-06-26T15:15:17Z

this would be great!
maybe a warning at import could be a good idea, our team develops on mac, and we run the LLM workloads on a server.
But it would be great to be able to install the project locally for debugging some of the helper code without having to maintain separate deps.

cdpierse · 2024-08-09T14:29:34Z

➕ 1 for this feature. It would be really nice to enable local development for code completion, and documentation references.

youkaichao · 2024-08-09T16:36:08Z

I think this feature is useful, but can you use the existing infra without introducing another env var?

e.g. VLLM_TARGET_DEVICE="empty" python setup.py develop

tomeras91 · 2024-08-11T12:48:49Z

requirements-cuda.txt

@@ -7,5 +7,5 @@ nvidia-ml-py # for pynvml package
 torch == 2.4.0
 # These must be updated alongside torch
 torchvision == 0.19   # Required for phi3v processor. See https://github.com/pytorch/vision?tab=readme-ov-file#installation for corresponding version
-xformers == 0.0.27.post2  # Requires PyTorch 2.4.0
-vllm-flash-attn == 2.6.1  # Requires PyTorch 2.4.0
+xformers == 0.0.27.post2; platform_system == 'Linux' and platform_machine == 'x86_64'  # Requires PyTorch 2.4.0


This change is because we take the CUDA requirements for the "empty" device wheel, and xformers and vllm-flash-attn are available only on Linux

platform_system == 'Linux' makes sense to me.

is platform_machine == 'x86_64' necessary?

vllm-flash-attn have published wheels only for x86_64, and no published tar.gz - https://pypi.org/project/vllm-flash-attn/#files
xformers also has wheels only for 64bit machines. It does have a tar.gz but from what I found online it can't be installed on 32bit - https://pypi.org/project/xformers/#files

So I'm pretty sure it's needed for both

tomeras91 · 2024-08-11T12:49:57Z

I think this feature is useful, but can you use the existing infra without introducing another env var?

e.g. VLLM_TARGET_DEVICE="empty" python setup.py develop

Thanks @youkaichao . Done now
also - the diff between this an main is very minimal due to the great work in #6786, solving the import issues with triton

tomeras91 · 2024-08-11T16:25:52Z

setup.py

-# vLLM only supports Linux platform
-assert sys.platform.startswith(
-    "linux"), "vLLM only supports Linux platform (including WSL)."
+if not sys.platform.startswith("linux"):


This change actually makes it possible to install the published tar.gz on mac, by setting VLLM_TARGET_DEVICE to "empty". Also logging a warning about vLLM not actually being able to run.

tomeras91 · 2024-08-11T16:26:16Z

setup.py

@@ -350,7 +356,9 @@ def find_version(filepath: str) -> str:
 def get_vllm_version() -> str:
    version = find_version(get_path("vllm", "version.py"))

-    if _is_cuda():
+    if _no_device():
+        version += "+empty"


I was actually not sure if it is better to add "+empty" to the version or not.. WDYT?

adding "+empty" looks good to me.

tomeras91 · 2024-08-11T16:29:30Z

@youkaichao - now this PR actually does 2 things:

It enables a local platform agnostic build of vLLM with VLLM_TARGET_DEVICE="empty" python setup.py
It actually allows build of the published tar.gz on macs. So now a developer working on mac can run pip install vllm and actually have vllm installed and importable (though not runnable)

IMO the second point is the nicer feature here. Really makes development using vLLM much easier on macs. I'm not sure though if you'd want to include it.. so if you think it's not a good idea, we can discard it and just leave the VLLM_TARGET_DEVICE="empty" option

youkaichao · 2024-08-11T18:47:50Z

@tomeras91 this feature is only for dev and debugging. I don't think it make any sense to publish to pypi. Making VLLM_TARGET_DEVICE="empty" python setup.py work is enough.

tomeras91 · 2024-08-11T19:24:28Z

@tomeras91 this feature is only for dev and debugging. I don't think it make any sense to publish to pypi. Making VLLM_TARGET_DEVICE="empty" python setup.py work is enough.

I understand and agree.
I wasn't suggesting to publish the "empty" wheel to pypi. The change here just makes it possible to install the already published tar.gz on mac

youkaichao

thanks for the great work! I manually verified that VLLM_TARGET_DEVICE="empty" pip install -vvv -e . works for macos now🎉

youkaichao · 2024-08-11T20:11:35Z

@tomeras91 can you add a followup pr for https://docs.vllm.ai/en/latest/getting_started/installation.html , to tell users how to use this feature?

tomeras91 · 2024-08-11T21:01:13Z

@tomeras91 can you add a followup pr for https://docs.vllm.ai/en/latest/getting_started/installation.html , to tell users how to use this feature?

Sure - here: #7403

…ect#4773)

tomeras91 added 5 commits May 12, 2024 16:02

mark xformers and vllm-flash-attn as installable only on x86_64 Linux

03850aa

changes in vllm to mock triton in case it can't be imported

f47bc9c

changes to setup.py to allow PLATFORM_AGNOSTIC_BUILD

bc83673

format and lint fixes

947c2fa

no need for triton installation in CPU requirements. Import errors ar…

24c513e

…e fixed

tomeras91 commented May 12, 2024

View reviewed changes

Merge branch 'main' into platform-agnostic-wheel

3c1819c

Merge branch 'vllm-project:main' into platform-agnostic-wheel

c68f261

tomeras91 added 3 commits August 11, 2024 12:49

Merge branch 'main' into platform-agnostic-wheel

b77ea4c

reduce diffs from main

cd0729f

implement using VLLM_TARGET_DEVICE='empty'

18b33ff

tomeras91 commented Aug 11, 2024

View reviewed changes

tomeras91 added 2 commits August 11, 2024 19:22

make build of .tar.gz possible on mac

f632c8e

Add "+empty" to version for build with VLLM_TARGET_DEVICE="empty"

37c40ed

tomeras91 commented Aug 11, 2024

View reviewed changes

(1) no f-strings in logs (2) warning instead of info log

cf75037

youkaichao approved these changes Aug 11, 2024

View reviewed changes

youkaichao changed the title ~~[CI/Build] Platform agnostic wheel~~ [CI/Build] build on empty device for better dev experience Aug 11, 2024

youkaichao merged commit 3860879 into vllm-project:main Aug 11, 2024
27 checks passed

tomeras91 mentioned this pull request Aug 11, 2024

[Doc] add instructions about building vLLM with VLLM_TARGET_DEVICE=empty #7403

Merged

tomeras91 deleted the platform-agnostic-wheel branch August 12, 2024 15:00

sfc-gh-mkeralapura pushed a commit to sfc-gh-mkeralapura/vllm that referenced this pull request Aug 12, 2024

[CI/Build] build on empty device for better dev experience (vllm-proj…

e46d6e0

…ect#4773)

kylesayrs pushed a commit to neuralmagic/vllm that referenced this pull request Aug 17, 2024

[CI/Build] build on empty device for better dev experience (vllm-proj…

91689d5

…ect#4773)

fialhocoelho pushed a commit to opendatahub-io/vllm that referenced this pull request Aug 22, 2024

[CI/Build] build on empty device for better dev experience (vllm-proj…

9585433

…ect#4773)

tomeras91 mentioned this pull request Sep 3, 2024

[CI/Build] fix: Add the +empty tag to the version only when the VLLM_TARGET_DEVICE envvar was explicitly set to "empty" #8118

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[CI/Build] build on empty device for better dev experience #4773

[CI/Build] build on empty device for better dev experience #4773

tomeras91 commented May 12, 2024

tomeras91 left a comment

tomeras91 May 12, 2024

simon-mo commented May 23, 2024 •

edited

Loading

tomeras91 commented May 23, 2024

hugolytics commented Jun 26, 2024

cdpierse commented Aug 9, 2024

youkaichao commented Aug 9, 2024

tomeras91 Aug 11, 2024

youkaichao Aug 11, 2024

tomeras91 Aug 11, 2024

tomeras91 commented Aug 11, 2024

tomeras91 Aug 11, 2024

tomeras91 Aug 11, 2024

youkaichao Aug 11, 2024

tomeras91 commented Aug 11, 2024

youkaichao commented Aug 11, 2024

tomeras91 commented Aug 11, 2024

youkaichao left a comment

youkaichao commented Aug 11, 2024

tomeras91 commented Aug 11, 2024

[CI/Build] build on empty device for better dev experience #4773

[CI/Build] build on empty device for better dev experience #4773

Conversation

tomeras91 commented May 12, 2024

tomeras91 left a comment

Choose a reason for hiding this comment

tomeras91 May 12, 2024

Choose a reason for hiding this comment

simon-mo commented May 23, 2024 • edited Loading

tomeras91 commented May 23, 2024

hugolytics commented Jun 26, 2024

cdpierse commented Aug 9, 2024

youkaichao commented Aug 9, 2024

tomeras91 Aug 11, 2024

Choose a reason for hiding this comment

youkaichao Aug 11, 2024

Choose a reason for hiding this comment

tomeras91 Aug 11, 2024

Choose a reason for hiding this comment

tomeras91 commented Aug 11, 2024

tomeras91 Aug 11, 2024

Choose a reason for hiding this comment

tomeras91 Aug 11, 2024

Choose a reason for hiding this comment

youkaichao Aug 11, 2024

Choose a reason for hiding this comment

tomeras91 commented Aug 11, 2024

youkaichao commented Aug 11, 2024

tomeras91 commented Aug 11, 2024

youkaichao left a comment

Choose a reason for hiding this comment

youkaichao commented Aug 11, 2024

tomeras91 commented Aug 11, 2024

simon-mo commented May 23, 2024 •

edited

Loading