Additional mask support on FA2 #57276
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add masked support on bwd. Unpadded kernel to be tested.
|
你的PR提交成功，感谢你对开源项目的贡献! |
|
✅ This PR's description meets the template requirements! |
Fix ci: PADDLE_ENFORCE format. Remove test case: return_softmax && dropout==0
umiswing
changed the title
Xreki
reviewed
Sep 22, 2023
| params.seed, | ||
| params.offset, | ||
| params.attn_mask_tensor ? params.attn_mask_tensor->data() : nullptr, | ||
| params.mask_dims.data()); |
There was a problem hiding this comment.
mask_dims传参这里，需要加上对params.attn_mask_tensor的判断吗？
There was a problem hiding this comment.
mask_dims传参这里，需要加上对params.attn_mask_tensor的判断吗？
fa2 capi里，会判断只有attn_mask不为nullptr时才访问mask_dims。所以功能上，不需要加判断了。不过需要注意的是，attn_mask_tensor为nullptr时，mask_dims是一个空的vector。c++标准没有定义empty vector的data()会返回什么，根据编译器实际的实现，可能是nullptr，也可能是任意值。
| auto rank = origin_dims.size(); | ||
| PADDLE_ENFORCE_GE( | ||
| rank, | ||
| 4, |
There was a problem hiding this comment.
如果rank本来就是4的话，那L68 - L75的维度变换就没有必要了。之前之所以加这个逻辑，是因为蛋白质里面的维度是5-D的，需要变换成4-D。
tianshuo78520a
approved these changes
Sep 22, 2023
AnnaTrainingG
pushed a commit
to AnnaTrainingG/Paddle
that referenced
this pull request
Sep 26, 2023
* Add addition mask support. Tested on FlashAttnKernel. * Fix bug in fwd (temporarily). Add masked support on bwd. Unpadded kernel to be tested. * Add unscale on padded kernel. * Add varlen mask. * Remove redundant compute_scale_q * Remove redundant comment. Fix ci: PADDLE_ENFORCE format. Remove test case: return_softmax && dropout==0 * Add mask type check. * Update submodules.
AnnaTrainingG
pushed a commit
to AnnaTrainingG/Paddle
that referenced
this pull request
Sep 26, 2023
* Add addition mask support. Tested on FlashAttnKernel. * Fix bug in fwd (temporarily). Add masked support on bwd. Unpadded kernel to be tested. * Add unscale on padded kernel. * Add varlen mask. * Remove redundant compute_scale_q * Remove redundant comment. Fix ci: PADDLE_ENFORCE format. Remove test case: return_softmax && dropout==0 * Add mask type check. * Update submodules.
Frida-a
pushed a commit
to Frida-a/Paddle
that referenced
this pull request
Oct 14, 2023
* Add addition mask support. Tested on FlashAttnKernel. * Fix bug in fwd (temporarily). Add masked support on bwd. Unpadded kernel to be tested. * Add unscale on padded kernel. * Add varlen mask. * Remove redundant compute_scale_q * Remove redundant comment. Fix ci: PADDLE_ENFORCE format. Remove test case: return_softmax && dropout==0 * Add mask type check. * Update submodules.
jiahy0825
pushed a commit
to jiahy0825/Paddle
that referenced
this pull request
Oct 16, 2023
* Add addition mask support. Tested on FlashAttnKernel. * Fix bug in fwd (temporarily). Add masked support on bwd. Unpadded kernel to be tested. * Add unscale on padded kernel. * Add varlen mask. * Remove redundant compute_scale_q * Remove redundant comment. Fix ci: PADDLE_ENFORCE format. Remove test case: return_softmax && dropout==0 * Add mask type check. * Update submodules.
AnnaTrainingG
pushed a commit
to AnnaTrainingG/Paddle
that referenced
this pull request
Nov 6, 2023
* Add addition mask support. Tested on FlashAttnKernel. * Fix bug in fwd (temporarily). Add masked support on bwd. Unpadded kernel to be tested. * Add unscale on padded kernel. * Add varlen mask. * Remove redundant compute_scale_q * Remove redundant comment. Fix ci: PADDLE_ENFORCE format. Remove test case: return_softmax && dropout==0 * Add mask type check. * Update submodules.
AnnaTrainingG
pushed a commit
to AnnaTrainingG/Paddle
that referenced
this pull request
Nov 6, 2023
* Add addition mask support. Tested on FlashAttnKernel. * Fix bug in fwd (temporarily). Add masked support on bwd. Unpadded kernel to be tested. * Add unscale on padded kernel. * Add varlen mask. * Remove redundant compute_scale_q * Remove redundant comment. Fix ci: PADDLE_ENFORCE format. Remove test case: return_softmax && dropout==0 * Add mask type check. * Update submodules.
danleifeng
pushed a commit
to danleifeng/Paddle
that referenced
this pull request
Nov 14, 2023
* Add addition mask support. Tested on FlashAttnKernel. * Fix bug in fwd (temporarily). Add masked support on bwd. Unpadded kernel to be tested. * Add unscale on padded kernel. * Add varlen mask. * Remove redundant compute_scale_q * Remove redundant comment. Fix ci: PADDLE_ENFORCE format. Remove test case: return_softmax && dropout==0 * Add mask type check. * Update submodules.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR types
Performance optimization
PR changes
OPs
Description
Pcard-70459
为更新Mask FA的支持，依赖PaddlePaddle/flash-attention#19 。