Basic text to numbers tokenizer for machine learning

Split text into tokens.

Splits a JSON string into an annotated list of tokens

Lexers, tokenizers, parsers, compilers, renderers, stringifiers... What's the difference, and how do they work?

Yet another powerful tokenizer in js.

A PHP Library to extract n-grams from a text. Simple preprocessing tools (cleaning, tokenizing) included.

Adds a location object to snapdragon token or AST node.

Snapdragon utility for creating a stack.

Create a snapdragon token. Used by the snapdragon lexer, but can also be used by plugins.

simple regex for correcting punctuations

Uses babel to extract JavaScript code comments from a string. Returns an array of comment objects, with line, column, index, comment type and comment string.

String tokenizer

A pythonic wrapper for Stanford CoreNLP.

Easily scan a string with an object of regex patterns to produce an array of tokens. ~100 sloc.

Extract JavaScript code comments from a string or glob of files.

textmining_project

Lazy calculations for Python based on iterators