grad_multiply function use?

[piyank22]

`class GradMultiply(torch.autograd.Function):
@staticmethod
def forward(ctx, x, scale):
  ctx.scale = scale
  res = x.new(x)
  ctx.mark_shared_storage((x, res))
  return res

@staticmethod
  def backward(ctx, grad):
  return grad * ctx.scale, None`

on reading the paper i have either completely missed or not come across something where we had to scale the gradient flowing back,
Please point out where i can get clarity on its use and why as to it has positive impact on the result.

[piyank22]

[https://github.com/facebookresearch/fairseq-py/blob/master/fairseq/models/fconv.py#L118](Line in fconv.py)

[gaopeng-eugene]

Why you scale the gradient ? Any explanation ?

[aa1607]

In case you were still wondering, it says here in the paper: "For convolutional decoders with multiple attention, we scale the gradients for the encoder layers by the number of attention mechanisms we use; we exclude source word embeddings. We found this to stabilize learning since the encoder received too much gradient otherwise."

https://arxiv.org/pdf/1705.03122.pdf