[Kernel] Update Cutlass fp8 configs #5144
Conversation
|
PFA a heatmap generated from the Gemm-Shape vs Cutlass-Op sweep done on an H100. Pointers on how to read the heatmap: Cutlass Op naming convention:
|
There was a problem hiding this comment.
LGTM.
One side question: If there's a new GPU coming out (e.g., B100), what would be the tuning process to update the configs? It's better to document this process somewhere so that we could leverage the community power to maintain the coverage in the future.
| @@ -0,0 +1,74 @@ | |||
| WEIGHT_SHAPES = { | |||
| "mistralai/Mistral-7B-v0.1/TP1": [ | |||
There was a problem hiding this comment.
A better way to format these shapes would be just having shapes for TP1. Then we could accept --tp-size in cutlass_benchmark.py to calculate the desired shape.
There was a problem hiding this comment.
I see, but then we need to store information on which dimension to divide. i.e.
"mistralai/Mistral-7B-v0.1/TP1": [
[4096, 6144],
[4096, 4096],
[4096, 28672],
[14336, 4096],
],
"mistralai/Mistral-7B-v0.1/TP2": [
[4096, 3072], # divide dim 2
[2048, 4096], # divide dim 1
[4096, 14336], # divide dim 2
[7168, 4096], # divide dim 1
],
perhaps we can store the divide dim as a third element in the list ?
There was a problem hiding this comment.
Maybe we could do this:
"mistralai/Mistral-7B": [
([4096, 6144], 1),
([4096, 4096], 0),
([4096, 28672], 1),
([14336, 4096], 0),
],
Basically separate the sharding dimension to another tuple element to reduce potential confusion.
There was a problem hiding this comment.
yeah. definitely better 👍
|
Thanks for the review @comaniac. |
| @@ -0,0 +1,355 @@ | |||
| import argparse | |||
There was a problem hiding this comment.
nit: renaming this to be more specific rather than the same name as the directory would be nice, such as w8a8_benchmarks.py
There was a problem hiding this comment.
Renamed 👍
|
|
||
| # cutlass impl | ||
| timers.append( | ||
| bench_fn(a, b, scale_a.to(device="cpu"), scale_b.to(device="cpu"), |
There was a problem hiding this comment.
Is it intentional to put the scales on the cpu?
There was a problem hiding this comment.
At the moment - yes. the scales are singleton tensors and the cutlass kernel interface will trigger a GPU-to-CPU copy if they aren't already on the CPU. We put the scales on the CPU so we don't time the copy.
This will change when #5137 lands.
|
btw, I have 2 questions about this kernel: 1.I'm also curious about the performance of batch size 2-15. Recall that we pad the inputs to achieve better performance of Thanks. |
Hey Cody - I am not sure about the impact padding has. I did not focus on that batch-size range. I ran a quick benchmark sweep on the H100 I am using and saw pretty negligible perf-delta between batch-sizes {1, 4, 8, 12, 16}. However, I am not sure if |
|
|
Thanks @comaniac - This the result of the experiment that I mentioned earlier, <style type="text/css"></style>
Were you seeing something similar for the pytorch numbers ? Also, let me know if you have some shapes for me to try. |
|
@comaniac - About,
The cutlass kernel interface does seem to have this limitation - Line 182 in e9899fb ping @tlrmchlsmth |
re-trigger CI
|
These kernels need N and K to be divisible by 16. In the GEMM kernel definitions, the strides are defined to be multiples of 16. (It's hard to see this from the code right now. I'll make a pass to make that part more legible) We definitely need some of these for performance on larger matrices but can look into lifting or relaxing for the MoE kernels |
Thanks for the numbers. So we can see that for PyTorch results, batch size 16 is always better than batch size < 16 for all workloads you benchmarked (for example, |
Co-authored-by: Varun Sundar Rabindranath <varun@neuralmagic.com> Co-authored-by: Robert Shaw <114415538+robertgshaw2-neuralmagic@users.noreply.github.com>
Switching from torch._scaled_mm to vLLM's cutlass fp8 kernels when supported as we are seeing 5-15% improvement in e2e performance on neuralmagic/Meta-Llama-3-8B-Instruct-FP8 see https://docs.google.com/spreadsheets/d/1GiAnmzyGHgZ6zL_LDSTm35Bdrt4A8AaFEurDlISYYA4/ for some quick e2e benchmarks and #5144 for comparisons across different GEMM sizes.
) Switching from torch._scaled_mm to vLLM's cutlass fp8 kernels when supported as we are seeing 5-15% improvement in e2e performance on neuralmagic/Meta-Llama-3-8B-Instruct-FP8 see https://docs.google.com/spreadsheets/d/1GiAnmzyGHgZ6zL_LDSTm35Bdrt4A8AaFEurDlISYYA4/ for some quick e2e benchmarks and vllm-project#5144 for comparisons across different GEMM sizes.
Co-authored-by: Varun Sundar Rabindranath <varun@neuralmagic.com> Co-authored-by: Robert Shaw <114415538+robertgshaw2-neuralmagic@users.noreply.github.com>
) Switching from torch._scaled_mm to vLLM's cutlass fp8 kernels when supported as we are seeing 5-15% improvement in e2e performance on neuralmagic/Meta-Llama-3-8B-Instruct-FP8 see https://docs.google.com/spreadsheets/d/1GiAnmzyGHgZ6zL_LDSTm35Bdrt4A8AaFEurDlISYYA4/ for some quick e2e benchmarks and vllm-project#5144 for comparisons across different GEMM sizes.
Co-authored-by: Varun Sundar Rabindranath <varun@neuralmagic.com> Co-authored-by: Robert Shaw <114415538+robertgshaw2-neuralmagic@users.noreply.github.com>
) Switching from torch._scaled_mm to vLLM's cutlass fp8 kernels when supported as we are seeing 5-15% improvement in e2e performance on neuralmagic/Meta-Llama-3-8B-Instruct-FP8 see https://docs.google.com/spreadsheets/d/1GiAnmzyGHgZ6zL_LDSTm35Bdrt4A8AaFEurDlISYYA4/ for some quick e2e benchmarks and vllm-project#5144 for comparisons across different GEMM sizes.
Co-authored-by: Varun Sundar Rabindranath <varun@neuralmagic.com> Co-authored-by: Robert Shaw <114415538+robertgshaw2-neuralmagic@users.noreply.github.com>
) Switching from torch._scaled_mm to vLLM's cutlass fp8 kernels when supported as we are seeing 5-15% improvement in e2e performance on neuralmagic/Meta-Llama-3-8B-Instruct-FP8 see https://docs.google.com/spreadsheets/d/1GiAnmzyGHgZ6zL_LDSTm35Bdrt4A8AaFEurDlISYYA4/ for some quick e2e benchmarks and vllm-project#5144 for comparisons across different GEMM sizes.
Co-authored-by: Varun Sundar Rabindranath <varun@neuralmagic.com> Co-authored-by: Robert Shaw <114415538+robertgshaw2-neuralmagic@users.noreply.github.com>
) Switching from torch._scaled_mm to vLLM's cutlass fp8 kernels when supported as we are seeing 5-15% improvement in e2e performance on neuralmagic/Meta-Llama-3-8B-Instruct-FP8 see https://docs.google.com/spreadsheets/d/1GiAnmzyGHgZ6zL_LDSTm35Bdrt4A8AaFEurDlISYYA4/ for some quick e2e benchmarks and vllm-project#5144 for comparisons across different GEMM sizes.
Co-authored-by: Varun Sundar Rabindranath <varun@neuralmagic.com> Co-authored-by: Robert Shaw <114415538+robertgshaw2-neuralmagic@users.noreply.github.com>
) Switching from torch._scaled_mm to vLLM's cutlass fp8 kernels when supported as we are seeing 5-15% improvement in e2e performance on neuralmagic/Meta-Llama-3-8B-Instruct-FP8 see https://docs.google.com/spreadsheets/d/1GiAnmzyGHgZ6zL_LDSTm35Bdrt4A8AaFEurDlISYYA4/ for some quick e2e benchmarks and vllm-project#5144 for comparisons across different GEMM sizes.
Co-authored-by: Varun Sundar Rabindranath <varun@neuralmagic.com> Co-authored-by: Robert Shaw <114415538+robertgshaw2-neuralmagic@users.noreply.github.com>
) Switching from torch._scaled_mm to vLLM's cutlass fp8 kernels when supported as we are seeing 5-15% improvement in e2e performance on neuralmagic/Meta-Llama-3-8B-Instruct-FP8 see https://docs.google.com/spreadsheets/d/1GiAnmzyGHgZ6zL_LDSTm35Bdrt4A8AaFEurDlISYYA4/ for some quick e2e benchmarks and vllm-project#5144 for comparisons across different GEMM sizes.
This PR:
The kernel configurations were selected from benchmark sweeps performed on H100 GPUs on a variety of commonly encountered GEMM shapes.
Numbers:
GPU H100:
Command:
python3 benchmarks/cutlass_benchmarks/cutlass_benchmarks.py --dtype fp8 model_bench --models meta-llama/Llama-2-7b-hf/TP1<style type="text/css"></style>
PR Checklist (Click to Expand)
Thank you for your contribution to vLLM! Before submitting the pull request, please ensure the PR meets the following criteria. This helps vLLM maintain the code quality and improve the efficiency of the review process.
PR Title and Classification
Only specific types of PRs will be reviewed. The PR title is prefixed appropriately to indicate the type of change. Please use one of the following:
[Bugfix]for bug fixes.[CI/Build]for build or continuous integration improvements.[Doc]for documentation fixes and improvements.[Model]for adding a new model or improving an existing model. Model name should appear in the title.[Frontend]For changes on the vLLM frontend (e.g., OpenAI API server,LLMclass, etc.)[Kernel]for changes affecting CUDA kernels or other compute kernels.[Core]for changes in the core vLLM logic (e.g.,LLMEngine,AsyncLLMEngine,Scheduler, etc.)[Hardware][Vendor]for hardware-specific changes. Vendor name should appear in the prefix (e.g.,[Hardware][AMD]).[Misc]for PRs that do not fit the above categories. Please use this sparingly.Note: If the PR spans more than one category, please include all relevant prefixes.
Code Quality
The PR need to meet the following code quality standards:
format.shto format your code.docs/source/if the PR modifies the user-facing behaviors of vLLM. It helps vLLM user understand and utilize the new features or changes.Notes for Large Changes
Please keep the changes as concise as possible. For major architectural changes (>500 LOC excluding kernel/data/config/test), we would expect a GitHub issue (RFC) discussing the technical design and justification. Otherwise, we will tag it with
rfc-requiredand might not go through the PR.What to Expect for the Reviews
The goal of the vLLM team is to be a transparent reviewing machine. We would like to make the review process transparent and efficient and make sure no contributor feel confused or frustrated. However, the vLLM team is small, so we need to prioritize some PRs over others. Here is what you can expect from the review process:
action-requiredlabel on the PR if there are changes required. The contributor should address the comments and ping the reviewer to re-review the PR.Thank You
Finally, thank you for taking the time to read these guidelines and for your interest in contributing to vLLM. Your contributions make vLLM a great tool for everyone!