We provide some basic reading material here. More detailed pointers can also be found in the slides. Feel free to contact us for more references.
Michael Repplinger , Lisa Beinborn , Willem Zuidema:
Vector-space models of words and sentences
Daniel Jurafsky and James H. Martin: Speech and Language Processing (2020)
Chapter 3: N-gram Language Models
Chapter 9: Deep Learning Architectures for Sequence Processing
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, Luke Zettlemoyer (2018):
Deep Contextualized Word Representations (ELMO)
Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova (2019):
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia Polosukhin (2017):
Attention is all you need (Transformer)
Tom B. Brown et al. (2020):
Language Models are Few-Shot Learners (GPT-3)
The original ELMO and Bert papers are hard to read. Jay Alammar provides very good conceptual introductions:
A very basic LSTM language model in Keras with Dutch training data
Kristina Gulordava: An exmple LSTM language model in pytorch
Chris McCormick: Introductory tutorials to BERT
Alexander Rush: The annotated Transformer
Samira Abnar, Lisa Beinborn, Rochelle Choenni, Willem Zuidema (Blackbox NLP 2019):
Blackbox Meets Blackbox: Representational Similarity and Stability Analysis of Neural Language Models and Brains
M. Giulianelli, J. Harding, F. Mohnert, D. Hupkes, W. Zuidema (Best Paper Blackbox NLP 2018):
Under the Hood: Using Diagnostic Classifiers to Investigate and Improve how Language Models Track Agreement Information
Hollenstein & Beinborn (ACL 2021):
Relative Importance in Sentence Processing
N. Hollenstein, F. Pirovano, C. Zhang, L. Jäger, L. Beinborn (NAACL 2021):
Multilingual Language Models Predict Human Reading Behavior
Nora Hollenstein, Maria Barrett, Lisa Beinborn (LiNCR 2020):
Towards best practices for leveraging human language processing signals for natural language processing
Lisa Beinborn, Samira Abnar, Rochelle Choenni (2019):
Robust Evaluation of Language–Brain Encoding Experiments