Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add partial RMSnorm "pRMSNorm" variation
This was described in the RMSNorm paper as being able to accomplish the same task as RMSNorm usually with only performing calculations on the first 6% of entries. This is because the average of the RMSNorm changes more slowly the more items are added, and the RMSNorm authors noted that the tokens they measured had around the same value.
- Loading branch information