[QUESTION]What is the reason why bfloat16 requires gradient accumulation and allreductions need to be completed in fp32 #502
Labels
stale
No activity in 60 days on issue or PR
Comments
|
Marking as stale. No activity in 60 days. |
|
Same question, anyone knows the reason? |
|
Marking as stale. No activity in 60 days. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hi,I am currently studying the Megatron framework. I noticed that bfloat16 in megatron requires gradient accumulation, and allreductions need to be completed in fp32. Then the gradient communicates in fp32 format.
However, fp16 requires gradient accumulation, and allreductions can be completed in fp16.Then the gradient communicates in fp16 format.
I want to know what are the special reasons for these two different calculation methods
Line 159-160 of the megatron/arguments. py file
The text was updated successfully, but these errors were encountered: