A grammar describes the syntax of a programming language, and might be defined in Backus-Naur form (BNF). A lexer performs lexical analysis, turning text into tokens. A parser takes tokens and builds a data structure like an abstract syntax tree (AST). The parser is concerned with context: does the sequence of tokens fit the grammar? A compiler is a combined lexer and parser, built for a specific grammar.
Persian NLP Toolkit
Solves basic Russian NLP tasks, API for lower level Natasha projects
한국어 자연어처리를 위한 파이썬 라이브러리입니다. 단어 추출/ 토크나이저 / 품사판별/ 전처리의 기능을 제공합니다.
An Integrated Corpus Tool With Multilingual Support for the Study of Language, Literature, and Translation
Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction, using word statistics from 2 big corpora (english Wikipedia, twitter - 330mil english tweets).
Python port of Moses tokenizer, truecaser and normalizer
A Japanese tokenizer based on recurrent neural networks
Bitextor generates translation memories from multilingual websites
Text2Text Language Modeling Toolkit
Text to sentence splitter using heuristic algorithm by Philipp Koehn and Josh Schroeder.
Text tokenization and sentence segmentation (segtok v2)
DadmaTools is a Persian NLP tools developed by Dadmatech Co.
phoneme tokenizer and grapheme-to-phoneme model for 8k languages
Simple multilingual lemmatizer for Python, especially useful for speed and efficiency
一个轻量且功能全面的中文分词器，帮助学生了解分词器的工作原理。MicroTokenizer: A lightweight Chinese tokenizer designed for educational and research purposes. Provides a practical, hands-on approach to understanding NLP concepts, featuring multiple tokenization algorithms and customizable models. Ideal for students, researchers, and NLP enthusiasts..
Fast bare-bones BPE for modern tokenizer training
aim to use JapaneseTokenizer as easy as possible
A tokenizer and sentence splitter for German and English web and social media texts.
AAAI 2025: Codec Does Matter: Exploring the Semantic Shortcoming of Codec for Audio Language Model
The code and models for "An Empirical Study of Tokenization Strategies for Various Korean NLP Tasks" (AACL-IJCNLP 2020)