This repo contains tutorials covering transformer models using PyTorch 1.4 and TorchText 0.5 using Python 3.6.
If you find any mistakes with the code, please do not hesitate to submit an issue. I welcome any feedback, positive or negative!
To install PyTorch, see installation instructions on the PyTorch website.
To install TorchText:
pip install torchtextWe'll also make use of spaCy to tokenize our data. To install spaCy, follow the instructions here making sure to install both the English and German models with:
python -m spacy download en
python -m spacy download de- 1 - Multi Head Attention
- 2 - Multi Head Attention using(einsum Notation)
- 3 - Multi Query Attention
- 4 - Multi Query Attention using (einsum Notation)
Here are some things I looked at while making these tutorials. Some of it may be out of date.