Oil Parser Generator Project

Back to Tasks Under NLNet Grant

This is an introduction to an important subproject of https://www.oilshell.org

Quick Description

We want the Oil expression parser to work in C++, to make it faster. It already works in Python.

In Python, the oil_lang/grammar_gen.py tool reads the grammar oil_lang/grammar.pgen2. Then it spits out a bunch of parse tables in Python's "marshal" format At runtime, the pgen2/ library reads it. The .

• Issue 594: Generate Parse Tables for pgen-native, and integrate it into oil-native. This is part of the translation to C++. Right now we only have a slow parser in Python for the Oil expression language.

Background

How to Parse Shell Like a Programming Language explains our parsing approach. This already works in Python:

$ bin/oil --ast-format text -n -c 'echo "hello $name"'
(command.Simple
  words: [
    (compound_word parts:[(Token id:Id.Lit_Chars span_id:0 val:echo)])
...

And it's already translated to C++:

$ _bin/cxx-dbg/osh_eval  -n -c 'echo "hello $name"'
(command.Simple
  words: [
    (compound_word parts:[(Token id:Id.Lit_Chars span_id:0 val:echo)])
...

This part does not use pgen2.

Relevant Files

• oil_lang/grammar_gen.py
• oil_lang/grammar.pgen2
• _devbuild/gen/grammar.marshal and _devbuild/gen/grammar_nt.py (non-terminals)
• oil_lang/expr_parse.py -- a wrapper for the generated parser
• The pgen2/ directory
  • parse.py and more

Related

• Oil Native Quick Start