Implementation of the LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens Paper

🧰 The AutoTokenizer that TikToken always needed -- Load any tokenizer with TikToken now! ✨

the small distributed language model toolkit; fine-tune state-of-the-art LLMs anywhere, rapidly

This project shows how to derive the total number of training tokens from a large text dataset from 🤗 datasets with Apache Beam and Dataflow.

Use custom tokenizers in spacy-transformers

Python script for manipulating the existing tokenizer.

Use Huggingface Transformer and Tokenizers as Tensorflow Reusable SavedModels

Megatron-LM/GPT-NeoX compatible Text Encoder with 🤗Transformers AutoTokenizer.

Package to align tokens from different tokenizations.

Byte Pair Encoding (BPE) tokenizer tailored for the Turkish language

ML Model designed to learn compositional structure of LEGO assemblies

Create prompts with a given token length for testing LLMs and other transformers text models.