You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository has been archived by the owner on Jul 7, 2023. It is now read-only.
In typical dot product attention, logit which is the input matrix of softmax supposed to be divided by square rooted temperature like the equation below.
However, in this code, logit is just divided with temperature without a square root. Is it correct or wrong? If it is correct, could you explain why you didn't add square root?
The text was updated successfully, but these errors were encountered:
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.
tensor2tensor/tensor2tensor/layers/area_attention.py
Line 415 in 5623deb
In typical dot product attention, logit which is the input matrix of softmax supposed to be divided by square rooted temperature like the equation below.
However, in this code, logit is just divided with temperature without a square root. Is it correct or wrong? If it is correct, could you explain why you didn't add square root?
The text was updated successfully, but these errors were encountered: