Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Misc] Support attention logits soft-capping with flash-attn #7022

Merged
merged 5 commits into from
Aug 1, 2024

Conversation

WoosukKwon
Copy link
Collaborator

@WoosukKwon WoosukKwon commented Aug 1, 2024

This PR adds support for attention logits soft-capping in the FlashAttention backend.

This is done by

  1. Moving logits_soft_cap from AttentionMetadata to AttentionImpl.
  2. Using vllm-flash-attn == 2.6.1 which added support for soft-capping.

Note that vllm-flash-attn v2.6.1 uses PyTorch 2.4.0, so the PR must be merged after #6951

This will hopefully resolve many of the issues in running Gemma2 models.

Copy link

github-actions bot commented Aug 1, 2024

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which consists a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of default ones by unblocking the steps in your fast-check build on Buildkite UI.

Once the PR is approved and ready to go, please make sure to run full CI as it is required to merge (or just use auto-merge).

To run full CI, you can do one of these:

  • Comment /ready on the PR
  • Add ready label to the PR
  • Enable auto-merge.

🚀

Copy link
Collaborator

@comaniac comaniac left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@WoosukKwon WoosukKwon added the ready ONLY add when PR is ready to merge/full CI is needed label Aug 1, 2024
@WoosukKwon WoosukKwon merged commit 805a8a7 into main Aug 1, 2024
59 of 63 checks passed
@WoosukKwon WoosukKwon deleted the fa-gemma2 branch August 1, 2024 20:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ready ONLY add when PR is ready to merge/full CI is needed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants