fix mask-tying to sequence length

[erip]

What does this PR do?

Fixes #655

This PR allows a class-level causal attention mask for fixed sequence-length tasks; otherwise it creates a new mask on-the-fly in the forward pass to disentangle the mask from sequence length which can vary between batches in tasks like MT.

Before submitting

• Did you have fun?
  • Make sure you had fun coding 🙃
• Did you read the contributor guideline?
• Was this discussed/approved via a Github issue? (no need for typos, doc improvements)
  • N/A
• Did you make sure to update the docs?
  • N/A
• Did you write any new necessary tests?
  • N/A
• Did you update the changelog? (if needed)
  • N/A

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

[erip]

This is mostly a quick smoke test (haven't tested locally yet --- running into issues with local dev). I think this needs a unit test and probably an inspection of benchmark performance to ensure the new branching doesn't impact speed.

[erip]

I think this is ready for review. It's a pretty small change, benchmarks look OK, and tests pass (locally). cc @danthe3rd @blefaudeux

[danthe3rd]

Hi @erip and thanks for your contribution :)
I'm not familiar with this code - maybe @blefaudeux can have a look, or @fmassa ?

[blefaudeux]

LGTM, I was probably the one writing the stateful masking, but thinking a bit more it's not obvious that it's a win. Deferring to @fmassa if something is wrong here, but out of principle looks good to me ! Thanks @erip

[fmassa]

LGTM, thanks!

As discussed in the previous comments, we really need to do some cleanup and refactoring on those APIs. But this is already a good improvement, thanks!