REPP tokenizer

[goodmami]

Make a module that reads .repp files and tokenizes strings, without firing up a parser. See here: http://moin.delph-in.net/ReppTop

[alvations]

+1 Reading .repp files will be a good first step =)

From Rebecca's code, reading the LISP .rpp setting files seems to involve some boost regex compilation (boost::make_u32regex). Does anyone know whether they achieve the same regexes as python re.compile? More questions on http://stackoverflow.com/questions/34048609/converting-c-boost-regexes-to-python-re-regexes

[goodmami]

The ReppTop page says the LKB uses a different regex engine, but both are Perl compatible (PCRE). Rather than Python's re module, the regex module seems like it has better Perl compatibility.

[goodmami]

Following up on engine choice: regex is more PCRE than re, but I am using re's internals (from the sre_parse module) to analyze the replacement patterns for characterization and string expansion purposes, so it's not as easy as just swapping out the engine.

Also note that there is python-pcre for actual PCRE, but it hasn't been updated in a while.

For now I will just use the regular re library and document its differences. Later I may try to minimally modify the patterns and replacement templates to be more compatible (e.g., changing \10 to \g<1>0 so it behaves like the C++ REPP).