Fully functional reader macros

[fjarri]

Currently Hy has some limited support of reader macros (quote, # etc). Is it planned (or even wanted) to add full support of reader macros similar to CL's set-macro-character? I think it will require either replacing the current lexer+parser pair with a custom reader in Python, or, alternatively, leaving the lexer+parser as the first stage that compiles the actual reader written in Hy.

[Foxboron]

Basically, thats what we have been thinking about. But we ain't really sure what to do. Hack rply into a suitable form? Replace everything (again)?
I have been messing to see if its possible to add new things into the rply instance while its running...but uh....easier said then done.

On 12/15/2014 02:52 AM, Bogdan Opanchuk wrote:

Currently Hy has some limited support of reader macros (quote, |#| etc). Is it planned (or even wanted) to add full support of reader macros similar to CL's |set-macro-character|? I think it will
require either replacing the current lexer+parser pair with a custom reader in Python, or, alternatively, leaving the lexer+parser as the first stage that compiles the actual reader written in Hy.

—
Reply to this email directly or view it on GitHub #722.

[fjarri]

Based on my very limited understanding of how parsing works, I don't think you can represent full Lisp reader with a lexer-parser pair. So, no amount of tinkering with rply will help.

Of the two solutions I mentioned, I'd prefer the latter one: keep rply as a parser for the "static" subset of Hy, and then adopt some existing reader implementation (for example, Clojure reader written in Clojure).

What also slightly worries me is that a full reader reads input stream character by character, as opposed to rply lexer which uses regexps on the full string, so there will be some performance drop (can't say how significant, though).

[agentultra]

I have a branch sitting around that has an almost complete parser-combinator for Hy written in Hy… it will just need some love to update it with the recent changes.

Once we have an actual reader (and not a parser-generator) implementing reader-macros, while not trivial, is not far fetched.

On Dec 15, 2014, at 12:39 AM, Bogdan Opanchuk notifications@github.com wrote:

Based on my very limited understanding of how parsing works, I don't think you can represent full Lisp reader with a lexer-parser pair. So, no amount of tinkering with rply will help.

Of the two solutions I mentioned, I'd prefer the latter one: keep rply as a parser for the "static" subset of Hy, and then adopt some existing reader implementation (for example, Clojure reader written in Clojure).

What also slightly worries me is that a full reader reads input stream character by character, as opposed to rply lexer which uses regexps on the full string, so there will be some performance drop (can't say how significant, though).

—
Reply to this email directly or view it on GitHub.

[scauligi]

Bumping this from the grave; I was fooling around and wrote a fairly straightforward reader/parser that reads one character at a time, but my initial prototype seems to actually run a bit faster than the current rply lexer/parser.

If there's interest, I can implement the full grammar in this thing and get a PR going; having this would allow us to:

• Get rid of rply (which has stagnated)
• Implement constructs like #_, #{...}, etc. as proper reader macros instead of special-cased grammar rules
• Allow for user-defined reader macros...?(!)
• Have well-bracketed f-strings*, which I could not for the life of me figure out how to implement with rply
• Have more control over / have more informative syntax errors
• And also probably still have a (minor) performance improvement over the current lexer/parser

*Well-bracketed f-strings as in: f"text {(expr 'with "embedded strings" (or f"other {fstrings}"))} more text"

[Kodiologist]

That definitely sounds interesting, although it would probably be better to use funcparserlib than write our own parser from scratch.

[scauligi]

I tried a parser in funcparserlib* (as well as parsita and parsimonious), and they all end up slower than the current rply implementation. I also tried lark, which was just about comparable to rply.

The custom parser still beats all of these in terms of performance, while also easily allowing for a modifiable reader table for reader macros.

*My funcparserlib implementation is parsing by character instead of tokenizing, since early tokenization is what blocks us from better f-strings, custom readers, etc

---

Results from parsing a ~3100 line file, min of 25 runs reported

custom         0.1829 s
lark           0.3227 s
hy (rply)      0.3610 s
parsimonious   0.4295 s
funcparserlib  0.5004 s
parsita        0.5667 s

[allison-casey]

hot dang scauligi this sounds really cool! do you have a fork we can look at?