Replace special forms with result macros #2104

Kodiologist · 2021-06-21T18:34:13Z

Closes #2046.

I've implemented macros that can return Result objects or AST, and converted all special operators into such macros. This change has little immediate consequence for most Hy users, but simplifies how Hy works somewhat, and opens the way to users hiding or replacing "special operators" with their own implementations, or writing new result macros.

I removed some tests that are currently breaking. I think @allison-casey has fixes for most of these; could you add that in? I'll take a hack again at what's left.

I haven't yet made docs or NEWS changes. Let's make sure we like the code first.

Things we'll want to consider after this is merged:

How the removal of special forms should affect The great core shakeup #1992. I was planning to remove all core macros, leaving only special forms, but it looks like instead we'll have no special forms, and core macros will pick up the slack.
How users can hide core macros and make their own result macros. This will probably call for a combination of code changes and more docs.

scauligi · 2021-06-21T19:51:00Z

Yo this is awesome! hy/compiler.py is so... clean now. I'll wait to review until after all the tests are back in?

I really dig the idea of Hy itself being a fairly light reader/compiler, and most of the core functionality being implemented as (reader/result/regular) macros through the same API that any user can tap into.

allison-casey · 2021-06-26T15:55:43Z

sorry long week i'll get those patches in today

Kodiologist · 2021-06-26T19:19:49Z

sorry long week i'll get those patches in today

It's okay. I haven't been paying much attention to Hy over the past few days, either.

this will raise an exception on macro expansion and not just when used in an invalid location. such that (hy.macroexpand '(except [foo Exception])) is no longer possible

Isn't that an invalid location?

allison-casey · 2021-06-26T19:41:16Z

yes, but previous behavior wouldn't make that assumption about a special form and just returned the tree untouched since special form compilation was further back in the compilation order. Its okay if that's the new desired behavior, but its a departure that should acknowledge and document since it requires extra checking from downstream code.

Kodiologist · 2021-06-26T19:43:33Z

Yeah, it seems reasonable enough to me, since special forms per se no longer exist.

allison-casey · 2021-06-26T19:52:26Z

i think its just as easy at listing under breaking changes that macroexpanding a bare (except ...) (or other result operator with similar behavior) expression is will raise a value error, since the result macros is mostly a user invisible change

allison-casey · 2021-06-26T21:30:45Z

these fix the walk, destructure, and reserved tests. there is the question of what to do about bad roots. they were tracked in extra.names before since they were retrievable from the compiler under _bad_roots, but now that that's gonna i've added a fake-special function to extra to track those forbidden forms. not sure if this is the way we want to do this, but i think its not really any less brittle than what we had before

allison-casey · 2021-06-26T21:41:09Z

this should be all the fixes @Kodiologist have I missed anything?

Kodiologist · 2021-06-27T13:09:13Z

Nope, that covers it, Allie. Thanks a lot. I've rebased to undo my test removals. fake-special doesn't seem necessary for defining names since names already looks at hy.core.macros.__macros__. Do you just want to provide it for the user's sake?

Kodiologist · 2021-06-27T13:11:00Z

Correction: they're already provided by (special), not hy.core.macros.__macros__.

allison-casey · 2021-06-27T13:23:51Z

your right that they're provided by hy.core.macros.__macros__ so fake-special can probably just be removed

Kodiologist · 2021-06-27T18:08:26Z

I've reorganized the edits, and replaced special with macros since there are no longer special forms.

Kodiologist · 2021-06-27T18:10:12Z

@scauligi All tests passing if you want to check it out now.

scauligi · 2021-07-01T16:57:48Z

Been a bit busy this week but taking a look now!

scauligi

Pyright tells me hy.compiler._module_file_source is not being used anywhere, so we can delete that function while we're removing stuff from hy/compiler.py.

Other unused imports we can clean up:

hy/importer.py: re, io, tempfile, hy_ast_compile_flags

hy/compiler.py

scauligi · 2021-07-01T16:51:50Z

hy/compiler.py

-                "Python parse error in '{}': {}".format(root, e))
-
-        return Result(stmts=o) if exec_mode else o
-
    @builds_model(Expression)
    def compile_expression(self, expr, *, allow_annotation_expression=False):


allow_annotation_expression is vestigial, as well as is_annotate_expression

Unremoved. Both appear to currently be necessary for compiling annotation forms.

Ah I see is_annotate_expression does appear necessary, but the keyword arg allow_annotation_expression is not longer needed; you should add a forbid for annotate in hy/core/macros.hy to replace what it used to be doing.

hy/compiler.py

hy/macros.py

scauligi · 2021-07-01T21:46:44Z

hy/macros.py

+                expr = hy_compiler.this
+                root = unmangle(expr[0])


I stuck an assert tuple(expr[1:]) == args here and it passed a run of pytest.
Can we change the result macro fn signatures so that we can call it e.g. fn(hy_compiler, name, args, *parse_tree) instead of needing to pass on expr? From what I can tell, the only reason we need expr is for source line position, which we can (and probably should) handle up in hy/compiler.py anyway. Then we wouldn't need to muck around with setting/retrieving compiler.this.

The macros need to set the source-line positions when they're constructing the python AST objects. By the time they've returned an AST object up the call stack, it's too late for subordinate AST objects to have their positions set correctly.

Would ast.fix_missing_locations() solve that problem?

Hmm, I think so. I would try an ast.fix_missing_locations() step after each macro-expansion that produces a Result or AST, with a temporary outermost AST object that has the location information to be copied.

My one concern is that once you can write ast.X(…), with no location information, in place of asty.X(expr, …), people working on Hy's core macros will be liable to mistakenly write ast.X(…) when they meant asty.X(some_model, …). There won't be a crash; nobody will notice anything amiss unless they see an error message with a position that's not as far into the parse tree as it should be. Do you think getting rid of the expr parameter and replacing a lot of asty.X(expr, …) with ast.X(…) is worth these potential bugs?

If the compiler calls sets the position on the result of macroexpand() from the original expr, then calls ast.fix_missing_locations(), then none of the core macros will even need to use asty at all, unless I'm missing something.

You'll still need to set locations explicitly in order to point to arguments of the macro, or subcomponents of the arguments, instead of the whole form.

Ah yes, that's a good point.
Hmm, I still would prefer a way of not needing to set extra state in the compiler, but I'll leave that for a later PR.

hy/macros.py

hy/models.py

hy/core/result_macros.py

hy/contrib/walk.hy

hy/core/result_macros.py

scauligi · 2021-07-01T22:46:01Z

Lot of (small) comments mainly because it's a big PR, but on the whole this looks dope! I'm excited for this to land 🎉

Kodiologist · 2021-07-02T13:13:06Z

Pyright tells me hy.compiler._module_file_source is not being used anywhere, so we can delete that function while we're removing stuff from hy/compiler.py.

Done.

hy/importer.py: re, io, tempfile, hy_ast_compile_flags

Removed.

Kodiologist · 2021-07-02T13:18:46Z

Ugh, my mistake; I forgot to git clean when I was testing locally, so I didn't notice I broke something. I'm working on it.

Kodiologist · 2021-07-02T13:49:14Z

@scauligi Okay, how's that?

Kodiologist · 2021-07-03T18:21:56Z

I updated the documentation (dumping a section on implementation details which is only going to get further out of date as we get to Hy 1.0), so this is ready to merge if you're okay with it, @scauligi.

Co-authored-by: Allison Casey <allie.jo.casey@gmail.com>

This commit consists mostly of cutting special-form methods of the compiler and pasting them over as macros. Likewise, a lot of private compiler methods have become helper functions. The following changes are made: - Replace `@special` with `@result_macro` - Replace `self` with `compiler`, and change some method calls to function calls accordingly - Remove some leading underscores in names - Adjust spacing and the order of definitions - Add Outshine headers for Emacs in comments - Add a module docstring

Now that there are no special forms. `names` has also been updated. Co-authored-by: Allison Casey <allie.jo.casey@gmail.com>

Co-authored-by: Allison Casey <allie.jo.casey@gmail.com>

Implementation details that aren't supposed to affect the user shouldn't be documented.

scauligi · 2021-07-03T20:34:21Z

I'm so hype for this 🎉

scauligi · 2021-07-03T20:40:20Z

Oh minor nit if there's still time: looks like pkgutil in hy/compiler.py is also unused now

Kodiologist · 2021-07-03T20:41:16Z

I'm afraid you're about 1 minute 30 seconds too late.

Kodiologist force-pushed the result-macros branch from 2d8ccd7 to 399469c Compare June 27, 2021 13:06

Kodiologist force-pushed the result-macros branch from 399469c to eadcfed Compare June 27, 2021 18:07

scauligi reviewed Jul 1, 2021

View reviewed changes

Kodiologist force-pushed the result-macros branch from eadcfed to fd62e31 Compare July 2, 2021 13:12

Kodiologist force-pushed the result-macros branch 2 times, most recently from 6dd1768 to 63822b2 Compare July 2, 2021 13:36

Kodiologist added 3 commits July 2, 2021 09:42

Remove some unused imports in hy.importer

c7b1e3e

Remove some unused bits from hy.compiler

5a99864

Allow macros to return a Result or AST

af841e5

Kodiologist force-pushed the result-macros branch 2 times, most recently from 0accd7e to 1e6459e Compare July 2, 2021 13:46

Kodiologist and others added 18 commits July 3, 2021 16:25

Add infrastructure for core result macros

d8d4e9c

Co-authored-by: Allison Casey <allie.jo.casey@gmail.com>

Move the function is_unpack to hy.models

3e9d017

Add pattern macros

5cac7a9

Use macroexpand_1 instead of macroexpand in some tests

b885698

In operator tests, allow another error type

281c072

Make an error-message test for macros more lax

89397b9

Mark some result macros as shadowing

36292fc

Add more tests of #* with Python operators

4492913

Clean up imports in hy.compiler

c7b2285

Remove obsolete special-form machinery

2b8522f

Update the docstring of hy.macros.require

6e64840

Replace hy.extra.reserved.special with macros

cfd9c83

Now that there are no special forms. `names` has also been updated. Co-authored-by: Allison Casey <allie.jo.casey@gmail.com>

Update let for the removal of special forms

0766a84

Co-authored-by: Allison Casey <allie.jo.casey@gmail.com>

Remove partly outdated docs of Hy internals

013f986

Implementation details that aren't supposed to affect the user shouldn't be documented.

Remove some docs references to special forms

af63526

Reorganize api.rst

49fc205

Update NEWS

d5c8428

Kodiologist force-pushed the result-macros branch from e541115 to d5c8428 Compare July 3, 2021 20:30

scauligi approved these changes Jul 3, 2021

View reviewed changes

Kodiologist merged commit 66e299f into hylang:master Jul 3, 2021

Kodiologist mentioned this pull request Jul 4, 2021

Perhaps make result macros convenient for users #2108

Closed

allison-casey mentioned this pull request Jul 4, 2021

Macro name shadowing/collisions and removal #1689

Closed

This was referenced Jul 5, 2021

document the hidden &compiler parameter in defmacro #1455

Closed

Introspecting -compile-table constructs. #1441

Closed

Kodiologist mentioned this pull request Aug 3, 2021

Turn require into a macro #1692

Closed

Kodiologist deleted the result-macros branch February 3, 2022 22:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Replace special forms with result macros #2104

Replace special forms with result macros #2104

Kodiologist commented Jun 21, 2021

scauligi commented Jun 21, 2021

allison-casey commented Jun 26, 2021

Kodiologist commented Jun 26, 2021

allison-casey commented Jun 26, 2021 •

edited

Loading

Kodiologist commented Jun 26, 2021

allison-casey commented Jun 26, 2021

allison-casey commented Jun 26, 2021

allison-casey commented Jun 26, 2021

Kodiologist commented Jun 27, 2021

Kodiologist commented Jun 27, 2021

allison-casey commented Jun 27, 2021

Kodiologist commented Jun 27, 2021

Kodiologist commented Jun 27, 2021

scauligi commented Jul 1, 2021

scauligi left a comment

scauligi Jul 1, 2021

Kodiologist Jul 2, 2021

Kodiologist Jul 2, 2021

scauligi Jul 3, 2021

scauligi Jul 1, 2021

Kodiologist Jul 2, 2021

scauligi Jul 3, 2021

Kodiologist Jul 3, 2021

scauligi Jul 3, 2021

Kodiologist Jul 3, 2021

scauligi Jul 3, 2021

scauligi commented Jul 1, 2021

Kodiologist commented Jul 2, 2021

Kodiologist commented Jul 2, 2021

Kodiologist commented Jul 2, 2021

Kodiologist commented Jul 3, 2021

scauligi commented Jul 3, 2021

scauligi commented Jul 3, 2021

Kodiologist commented Jul 3, 2021

Replace special forms with result macros #2104

Replace special forms with result macros #2104

Conversation

Kodiologist commented Jun 21, 2021

scauligi commented Jun 21, 2021

allison-casey commented Jun 26, 2021

Kodiologist commented Jun 26, 2021

allison-casey commented Jun 26, 2021 • edited Loading

Kodiologist commented Jun 26, 2021

allison-casey commented Jun 26, 2021

allison-casey commented Jun 26, 2021

allison-casey commented Jun 26, 2021

Kodiologist commented Jun 27, 2021

Kodiologist commented Jun 27, 2021

allison-casey commented Jun 27, 2021

Kodiologist commented Jun 27, 2021

Kodiologist commented Jun 27, 2021

scauligi commented Jul 1, 2021

scauligi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

scauligi commented Jul 1, 2021

Kodiologist commented Jul 2, 2021

Kodiologist commented Jul 2, 2021

Kodiologist commented Jul 2, 2021

Kodiologist commented Jul 3, 2021

scauligi commented Jul 3, 2021

scauligi commented Jul 3, 2021

Kodiologist commented Jul 3, 2021

allison-casey commented Jun 26, 2021 •

edited

Loading