Merge pull request #2242 from allison-casey/custom-parser

Custom Parser and Reader Macros

NEWS.rst

@@ -3,6 +3,14 @@
 Unreleased
 ==============================
 
+Removals
+------------------------------
+* Tag macros have been removed. Use reader macros instead, rewriting
+  `(defmacro "#foo" [arg] …)` as
+  `(defreader foo (setv arg (.parse-one-form &reader)) …)`.
+* `hy.read-str` has been removed. Use `hy.read`, which now accepts
+  strings, instead.
+
 Breaking Changes
 ------------------------------
 * `if` now requires all three arguments. For cases with less than
@@ -13,6 +21,16 @@ Breaking Changes
      (cond [a b] [x y z])     ; Old
      (cond  a b  x (do y z))  ; New
 
+* `@#` is now a reader macro and has been moved to Hyrule.
+* `defmacro` once again requires the macro name as a symbol, not
+  a string literal.
+* The parser has been completely rewritten. It is mostly
+  backwards-compatible, with two exceptions:
+
+  - Unescaped double quotes are now allowed inside replacement
+    fields of f-strings.
+  - A bare `#` is no longer a legal symbol.
+
 * The mangling rules have been refined to account for Python's
   treatment of distinct names as referring to the same variable if
   they're NFKC-equivalent. Very little real code should be affected.
@@ -33,6 +51,7 @@ Bug Fixes
 
 New Features
 ------------------------------
+* Added user-defined reader macros, defined with `defreader`.
 * Python reserved words are allowed once more as parameter names and
   keyword arguments. Hy includes a workaround for a CPython bug that
   prevents the generation of legal Python code for these cases

docs/api.rst

@@ -334,20 +334,6 @@ base names, such that ``hy.core.macros.foo`` can be called as just ``foo``.
         => (infix (1 + 1))
         2
 
-   The name of the macro can be given as a string literal instead of a symbol. If the name starts with `#`, the macro can be called on a single argument without parentheses; such a macro is called a tag macro.
-
-     ::
-
-        => (defmacro "#x2" [form]
-        ...  `(do ~form ~form))
-
-     ::
-
-        => (setv foo 1)
-        => #x2 (+= foo 1)
-        => foo
-        3
-
 .. hy:function:: (if [test then else])
 
    ``if`` compiles to an :py:keyword:`if` expression (or compound ``if`` statement). The form ``test`` is evaluated and categorized as true or false according to :py:class:`bool`. If the result is true, ``then`` is evaluated and returned. Othewise, ``else`` is evaluated and returned.
@@ -1088,13 +1074,17 @@ base names, such that ``hy.core.macros.foo`` can be called as just ``foo``.
 
 .. hy:function:: (require [#* args])
 
-   ``require`` is used to get macros from one or more given modules. It allows
-   parameters in all the same formats as ``import``. ``require`` imports each
-   named module and then makes each requested macro available in the current
-   module.
+   ``require`` is used to import macros and reader macros from one or more given
+   modules. It allows parameters in all the same formats as ``import``.
+   ``require`` imports each named module and then makes each requested macro
+   available in the current module.
 
    The following are all equivalent ways to call a macro named ``foo`` in the
-   module ``mymodule``::
+   module ``mymodule``.
+
+   :strong:`Examples`
+
+   ::
 
        (require mymodule)
        (mymodule.foo 1)
@@ -1111,6 +1101,38 @@ base names, such that ``hy.core.macros.foo`` can be called as just ``foo``.
        (require mymodule [foo :as bar])
        (bar 1)
 
+   Reader macros are required using ``:readers [...]``.
+   The ``:macros`` kwarg can be optionally added for readability::
+
+       => (require mymodule :readers [!])
+       => (require mymodule [foo] :readers [!])
+       => (require mymodule :readers [!] [foo])
+       => (require mymodule :macros [foo] :readers [!])
+
+   Do note however, that requiring ``:readers``, but not specifying any regular
+   macros, will not bring that module's macros in under their absolute paths::
+
+       => (require mymodule :readers [!])
+       => (mymodule.foo)
+       Traceback (most recent call last):
+         File "stdin-cd49eaaabebc174c87ebe6bf15f2f8a28660feba", line 1, in <module>
+           (mymodule.foo)
+       NameError: name 'mymodule' is not defined
+
+   Unlike requiring regular macros, reader macros cannot be renamed
+   with ``:as``, and are not made available under their absolute paths
+   to their source module::
+
+      => (require mymodule :readers [!])
+      HySyntaxError: ...
+
+      => (require mymodule :readers [! :as &])
+      HySyntaxError: ...
+
+      => (require mymodule)
+      => mymodule.! x
+      NameError: name 'mymodule' is not defined
+
    To define which macros are collected by ``(require mymodule *)``, set the
    variable ``_hy_export_macros`` (analogous to Python's ``__all__``) to a list
    of :ref:`mangled <mangling>` macro names, which is accomplished most
@@ -1593,10 +1615,12 @@ Hy
 The ``hy`` module is auto imported into every Hy module and provides convient access to
 the following methods
 
-.. hy:autofunction:: hy.read-str
-
 .. hy:autofunction:: hy.read
 
+.. hy:autofunction:: hy.read_many
+
+.. hy:autofunction:: hy.read_module
+
 .. hy:autofunction:: hy.eval
 
 .. hy:autofunction:: hy.repr
@@ -1617,6 +1641,23 @@ the following methods
 
 .. hy:autofunction:: hy.as-model
 
+.. _reader-macros:
+
+Reader Macros
+-------------
+
+Like regular macros, reader macros should return a Hy form that will then be
+passed to the compiler for execution. Reader macros access the Hy reader using
+the ``&reader`` name. It gives access to all of the text- and form-parsing logic
+that Hy uses to parse itself. See :py:class:`HyReader <hy.lex.hy_reader.HyReader>` and
+its base class :py:class:`Reader <hy.lex.reader.Reader>` for details regarding
+the available processing methods.
+
+.. autoclass:: hy.lex.hy_reader.HyReader
+   :members: parse, parse_one_form, parse_forms_until, read_default, fill_pos
+
+.. autoclass:: hy.lex.reader.Reader
+   :members:
 
 Python Operators
 ----------------

docs/language/internals.rst

@@ -37,7 +37,7 @@ Compound Models
 ---------------
 
 Parenthesized and bracketed lists are parsed as compound models by the
-Hy parser.
+Hy reader.
 
 Hy uses pretty-printing reprs for its compound models by default.
 If this is causing issues,
@@ -124,7 +124,7 @@ which compile down to unicode string literals (``str``) in Python.
 ``String``\s are immutable.
 
 Hy literal strings can span multiple lines, and are considered by the
-parser as a single unit, respecting the Python escapes for unicode
+reader as a single unit, respecting the Python escapes for unicode
 strings.
 
 ``String``\s have an attribute ``brackets`` that stores the custom

docs/language/interop.rst

@@ -113,13 +113,13 @@ Evaluating strings of Hy code from Python
 -----------------------------------------
 
 Evaluating a string (or ``file`` object) containing a Hy expression requires
-two separate steps. First, use the ``read_str`` function (or ``read`` for a
-``file`` object) to turn the expression into a Hy model:
+two separate steps. First, use the ``read`` function to turn the expression
+into a Hy model:
 
 .. code-block:: python
 
     >>> import hy
-    >>> expr = hy.read_str("(- (/ (+ 1 3 88) 2) 8)")
+    >>> expr = hy.read("(- (/ (+ 1 3 88) 2) 8)")
 
 Then, use the ``hy.eval`` function to evaluate it:
 

docs/language/repl.rst

@@ -8,7 +8,7 @@ Hy's REPL (read-eval-print loop) [#]_ functionality is implemented in the ``hy.c
 
 From a high level, a single cycle of the REPL consists of the following steps:
 
-1. tokenize and parse input with ``hy.lex.hy_parse``, generating Hy AST [#]_;
+1. read input with ``hy.lex.read_module``, generating Hy AST [#]_;
 2. compile Hy AST to Python AST with ``hy.compiler.hy_compile`` [#]_;
 3. execute the Python code with ``eval``; and
 4. print output, formatted with ``output_fn``.

docs/tutorial.rst

@@ -310,17 +310,6 @@ and macros that make it easy to construct forms programmatically, such as
 chapter has :ref:`a simple example <do-while>` of using ````` and ``~`` to
 define a new control construct ``do-while``.
 
-Sometimes it's nice to be able to call a one-parameter macro without
-parentheses. Tag macros allow this. The name of a tag macro is often just one
-character long, but since Hy allows most Unicode characters in the name of a
-macro (or ordinary variable), you won't out of characters soon. ::
-
-  => (defmacro "#↻" [code]
-  ...  (setv op (get code -1) params (list (cut code -1)))
-  ...  `(~op ~@params))
-  => #↻(1 2 3 +)
-  6
-
 What if you want to use a macro that's defined in a different module?
 ``import`` won't help, because it merely translates to a Python ``import``
 statement that's executed at run-time, and macros are expanded at compile-time,
@@ -332,6 +321,28 @@ which imports the module and makes macros available at compile-time.
    => (tutorial.macros.rev (1 2 3 +))
    6
 
+Hy also supports reader macros, which are similar to ordinary macros, but
+operate on raw source text rather than pre-parsed Hy forms. They can choose how
+much of the source code to consume after the point they are called, and return
+any code. Thus, reader macros can add entirely new syntax to Hy. For example,
+you could add a literal notation for Python's :class:`decimal.Decimal` class
+like so::
+
+    => (import  decimal [Decimal]  fractions [Fraction])
+    => (defreader d
+    ...   (.slurp-space &reader)
+    ...   `(Decimal ~(.read-ident &reader)))
+    => (print (repr #d .1))
+    Decimal('0.1')
+    => (print (Fraction #d .1))
+    1/10
+    => ;; Contrast with the normal floating-point .1:
+    => (print (Fraction .1))
+    3602879701896397/36028797018963968
+
+``require`` can pull in a reader macro defined in a different module with
+syntax like ``(require mymodule :readers [d])``.
+
 Hyrule
 ======
 

hy/__init__.py

@@ -22,7 +22,8 @@ def _initialize_env_var(env_var, default_val):
 
 _jit_imports = dict(
     read="hy.lex",
-    read_str="hy.lex",
+    read_many="hy.lex",
+    read_module="hy.lex",
     mangle="hy.lex",
     unmangle="hy.lex",
     eval=["hy.compiler", "hy_eval"],