v1.3.1

This is a minor update to fix issues with running the Colab on Prompt Engineering with Sequence Salience.

v1.3

This release updates how the Learning Interpretability Tool (LIT) can be deployed on Google Cloud. You can now use LIT to interpret foundation models—including Gemini, Gemma, Llama, and Mistral—using LIT's prompt debugging workflows. LIT now provides public container images to make it easier to deploy on your hosting platform of choice, with an updated tutorial for deploying LIT with Cloud Run.

New Stuff

• LIT on GCP -
  1075325,
  1acc868,
  55bfc99,
  180f68a,
  64114d5,
  2488aa7,
  9baac29,
  60bdc7c,
  7681476,
  4c81182,
  4e5e8e2,
  b9a0b82,
  424adce,
  1d019c7,
  f4436a2,

Non-breaking Changes, Bug Fixes, and Enhancements

• Upgrade LIT to MobX v6. - c1f5055
• Fix indexing issue in Sequence Salience module. - 58b1d2
• Load multiple model wrappers with shared model. - ba4d975
• Add the custom model and dataset loaders to prompt debugging notebook. - 338c6b
• Convert hosted demos images to multi-stage builds. - 4bf1f8
• Adding testing instructions to README. - f24b841
• More LIT documentation updates. - 2e9d267

v1.2

This release covers clean-ups on various obsolete demos, as well as improved
packaging and isolated dependencies on the GLUE, Penguin, Prompt Debugging with
Sequence Salience and TyDi demos for easier launch.

New Stuff

• Improved packaging and instructions for launching Prompt Debugging with
  Sequence Salience demo, as well as minor bug fixes -
  08289df
  675ca2d
  15eccb1
  e0e35c3
  c7970fb
  cee3b58

• Clean up of obsolete demos -
  b16059f
  f4c0990
  6aa2eb6
  c2fb41b
  dd196e9
  72fd772
  71d88fb
  aa49340
  fc7b0d0
  2475b3b
  a59641c
  1ed82d4
  7d5ef58
  992823b
  3dad2b0
  0656386
  27d7a84
  8863019
  71cbdba
  416d573

• Python requirements update and isolated setup for individual demos -
  bcc481e
  bb29f43
  fbd8874
  b3c120b
  5188c8c
  5639e3b

• Documentation cleanup and updates -
  afd51fe
  7dda659
  79ada6e
  1c8d6a0
  2e9d267

Non-breaking Changes, Bug Fixes, and Enhancements

• Refactor DataService reactions - 483082d
• Add warm_start option to LitWidget - a5265a4
• Pretty-printing of Model objects - 4fb3bde
• Avoid equivalent shuffles in Scrambler - 0d8c0d9
• Updated gunicorn config for demos running in Docker - b14e3b1
• Disable embeddings for TyDi - 7ff377f
• Cast embeddings to float32 before computing distances - 5456011
• Update colab examples to include installation of the lit-nlp package - 48b029c

Full Changelog: v1.1.1...v1.2

v1.1.1

This release covers various improvements for sequence salience, including new features in the UI module, support of more LLMs, and detailed tutorial and documentation on how to use the sequence salience module for prompt engineering.

New stuff

• New features in the sequence salience UI module - 62f18b2, f0417c9, fe5a705, 1ec8626, 15184a1, 84af141, 27cafd8, 3591e61, d108b59, 309c4f2, 99821d3, c8ee224

• Support of more models (GPT2, Gemma, Llama2, Mistral) on deep learning frameworks (Tensorflow, Pytorch) for Keras and Hugging Face - b26256a, 45887d3, b9941ed, 5ee7064, 8ea325b

• A tutorial to use sequence salience at our website and documentation updates - 962faaa, 96eff29, f731e6d, f4d7cac, 49e7736

Non-breaking Changes, Bug Fixes, and Enhancements

• Py typing fix - d70e3d3
• Improvements on the curves UI module - 3d61a09, 2efe62b
• Support model-column search in LIT Data Table - 525bf5e
• Obsolete code cleanup - 82abec6

v1.1

This release provides the capabilities to interpret and debug the behaviors of Generative AI models in LIT. Specifically, we added sequence salience, which explains the impact of the preceding tokens on the generated tokens produced by the GenAI models. Major changes include:

• An LM salience module in the LIT UI that computes generations, tokenization, and sequence salience on-demand;
• Computation of sequence salience at different granularities, from the smallest possible level of tokens, to more interpretable larger spans, such as words, sentences, lines, or paragraphs.
• Support of OSS modeling frameworks, including KerasNLP and Hugging Face Transformers for sequence salience computation.

This release would not have been possible without the work of our contributors. Many thanks to Ryan Mullins, Ian Tenney, Bin Du, and Cibi Arjun.

New Stuff

• LM salience module in the LIT UI - ab294bd, 5cffc4d, 40bb57a, d3980cc, 406fbc7, 77583e7, a758f98
• Sequence salience for decoder-only LM, with support for GPT-2 and KerasNLP - 27e6901, 80cf699, 1df3ba8, b6ab352, c97a710
• Prompt examples for sequence salience - 4f19891, 000c844, 34aa110, ca032ff

Non-breaking Changes, Bug Fixes, and Enhancements

• Improvements to display various fields and their default ranges - 8a3f366, e63b674, d274508
• Allow only displaying the UI layouts provided by users - a219863
• Internal dependency changes -, f254fa8, 724bdee, 2138bd9
• Fix issues with adding more than one example from counterfactual generators - d4302bd
• Fix issues with loading SimpleSentimentModel - ac8ed59
• Notebook widget improvements - cdf79eb
• Docs updates

v1.0.2

This is a minor update that fixes an issue with LIT failing to render in Colab.

v1.0.1

This is a major release, covering many new features and API changes from the dev branch since the v0.5 release over 8 months ago. This release includes a variety of breaking changes meant to simplify various aspects of the LIT API and visual changes to improve usability. This release includes over 250 commits. Major changes include:

• Refactored python code to remove _with_metadata methods from all component and model classes.
• Refactored Model and BatchedModel python classes to remove predict_minibatch method.
• Reworked UI and backend logic for dynamic loading of new datasets and models from the UI. This makes use of the new init_spec methods for datasets and model classes.
  • Added a blank demo with no models or datasets preloaded which allows for dynamic loading of models and datasets through the UI.
• Refactored to upgrade metrics calculation from a type of interpreter to its own top-level concept.
• Updated front-end layout code to default to a new layout that includes a full height side-panel on the left side to complement the existing top and bottom panels, providing for more customization of module layouts.
• Added automatic metrics calculations for multilabel models.
• Added target selector dropdown for saliency methods.
• A visual redesign of the Salience Clustering module.
• Improved searching capabilities in the Data Table module.
• Improved the Data Table module's display of long strings through a "Show more" capability.
• Updated to Python 3.10.
• Updated to Node 18 and Typescript 5.0.
• Improved documentation pages, now at https://pair-code.github.io/lit/documentation/

This release would not have been possible without the work of our new contributors in 2023. Many thanks to Minsuk Kahng, Nada Hussein, Oscar Wahltinez, Bin Du, and Cibi Arjun for your support and contributions to this project! A full list of contributors to this repo can be found at https://github.com/PAIR-code/lit/graphs/contributors.

v0.5

This is a major release, covering many new features from the dev branch since the v0.4 release nearly 11 months ago. Most notably, we're renaming! It's still LIT, but now the L stands for "Learning" instead of "Language", to better reflect the scope of LIT and support for non-text modalities like images and tabular data.

Additionally, we've made lots of improvements, including:

• New modules including salience clustering, tabular feature attribution, and a new Dive module for data exploration (inspired by our prior work on Facets Dive).
• New demos and tutorials for input salience comparison and tabular feature attribution.
• Many UI improvements, with better consistency across modules and shared functionality for colors, slicing, and faceting of data.
• Better performance on large datasets (up to 100k examples), as well as improvements to the type system and new validation routines (--validate) for models and datasets.
• Download data as CSV directly from tables in the UI, and in notebook mode access selected examples directly from Python.
• Update to Python 3.9 and TypeScript 4.7.

This release would not have been possible without the work of many new contributors in 2022. Many thanks to Crystal Qian, Shane Wong, Anjishnu Mukherjee, Aryan Chaurasia, Animesh Okhade, Daniel Levenson, Danila Sinopalnikov, Deepak Ramachandran, Rebecca Chen, Sebastian Ebert, and Yilei Yang for your support and contributions to this project!

v0.4.1

LIT version 0.4.1

This is a bug fix release aimed at improving visual clarity and common
workflows.

The UI has been slightly revamped, bugs have been fixed, and new capabilities
have been added. Notable changes include:

• Adds "open in new tab" feature to LIT Notebook widget
• Adds support for SparseMultilabelPreds to LIME
• Improves color consistency across the UI
• Switching NumPy instead of SciKit Learn for PCA
• Ensuring all built-in demos are compatible with the Docker
• Updating the Dockerfile to support run-time DEMO_NAME and DEMO_PORT args
• Fixed a rendering bug in the Confusion Matrix related column and row spans
  when "hide empty labels" is turned on

v0.4

LIT version 0.4.

The UI has been slightly revamped, bugs have been fixed, and new capabilities have been added. Notable changes include:

• Support for Google Cloud Vertex AI notebooks.
• Preliminary support for tabular and image data, in addition to NLP models.
• Addition of TCAV global interpretability method.
• New counterfactual generators for ablating or flipping text tokens for minimal changes to flip predictions.
• New counterfactual generator for tabular data for minimal changes to flip predictions.
• Partial depdence plots for tabular input features.
• Ability to set binary classification thresholds separately for different facets of the dataset
• Controls to find optimal thresholds across facets given different fairness constraints, such as demographic parity or equal opportunity.