Attention Visualizer

A fun project that turns out as a python package with a simple UI to visualize the self-attention score using the RoBERTa library. It is implemented for IPython Notebook environment with options to ignore tokens like "BOS", "[dot]s", or "stopwords". You can also look at a range or specific Layers/Heads.

Demo

Run the library on a Google Colab instance using the following link.

⚠️ There is a bug (only happens in Google Colab) that the instance's memory usage will increase everytime a new request is sent. Would be happy to know what the problem could be if anyone encounter it before.

Installation

The package is only hosted on Github for now. You can use pip to install the package.

  pip install git+https://github.com/AlaFalaki/AttentionVisualizer.git

Usage/Examples

Run the code below in an IPython Notebook.

import AttentionVisualizer as av

obj = av.AttentionVisualizer()
obj.show_controllers(with_sample=True)

Requirements

The package will automatically installs all the requirements.

• pytorch
• transformers
• ipywidgets
• NLTK

Implementation Details

If you are interested in the project and want to know more, I wrote a blog post on medium that explain the implementation in detail.

Attention Visualizer Package: Showcase Highest Scored Words Using RoBERTa Model

Citation

arXiv preprint: Attention Visualizer Package: Revealing Word Importance for Deeper Insight into Encoder-Only Transformer Models

@article{falaki2023attention,
  title={Attention Visualizer Package: Revealing Word Importance for Deeper Insight into Encoder-Only Transformer Models},
  author={Falaki, Ala Alam and Gras, Robin},
  journal={arXiv preprint arXiv:2308.14850},
  year={2023}
}