PEP 646: Explain the results of the new grammar/compiler change (#2189)

pep-0646.rst

@@ -871,9 +871,7 @@ shape properties of numerical computing programs.
 Grammar Changes
 ===============
 
-This PEP requires two grammar changes. Full diffs of ``python.gram``
-and simple tests to confirm correct behaviour are available at
-https://github.com/mrahtz/cpython/commits/pep646-grammar.
+This PEP requires two grammar changes.
 
 Change 1: Star Expressions in Indexes
 -------------------------------------
@@ -1015,21 +1013,58 @@ Where:
 
     star_annotation: ':' star_expression
 
-This accomplishes the desired outcome (making ``*args: *Ts`` not be a syntax
-error) while matching the behaviour of star-unpacking in other contexts:
-at runtime, ``__iter__`` is called on the starred object, and a tuple
-containing the items of the resulting iterator is set as the type annotion
-for ``args``. In other words, at runtime ``*args: *foo`` is equivalent to
-``*args: tuple(foo)``.
+We also need to deal with the ``star_expression`` that results from this
+construction. Normally, a ``star_expression`` occurs within the context
+of e.g. a list, so a ``star_expression`` is handled by essentially
+calling ``iter()`` on the starred object, and inserting the results
+of the resulting iterator into the list at the appropriate place. For
+``*args: *Ts``, however, we must process the ``star_expression`` in a
+different way.
+
+We do this by instead making a special case for the ``star_expression``
+resulting from ``*args: *Ts``, emitting code equivalent to
+``[annotation_value] = [*Ts]``. That is, we create an iterator from
+``Ts`` by calling ``Ts.__iter__``, fetch a single value from the iterator,
+verify that the iterator is exhausted, and set that value as the annotation
+value. This results in the unpacked ``TypeVarTuple`` being set directly
+as the runtime annotation for ``*args``:
 
 ::
 
     >>> Ts = TypeVarTuple('Ts')
-    >>> def foo(*args: *Ts): pass  # Equivalent to `*args: tuple(Ts)`
+    >>> def foo(*args: *Ts): pass
     >>> foo.__annotations__
-    {'args': (*Ts,)}
+    {'args': *Ts}
     # *Ts is the repr() of Ts._unpacked, an instance of UnpackedTypeVarTuple
 
+This allows the runtime annotation to be consistent with an AST representation
+that uses a ``Starred`` node for the annotations of ``args`` - in turn important
+for tools that rely on the AST such as mypy to correctly recognise the construction:
+
+::
+
+    >>> print(ast.dump(ast.parse('def foo(*args: *Ts): pass'), indent=2))
+    Module(
+      body=[
+        FunctionDef(
+          name='foo',
+          args=arguments(
+            posonlyargs=[],
+            args=[],
+            vararg=arg(
+              arg='args',
+              annotation=Starred(
+                value=Name(id='Ts', ctx=Load()),
+                ctx=Load())),
+            kwonlyargs=[],
+            kw_defaults=[],
+            defaults=[]),
+          body=[
+            Pass()],
+          decorator_list=[])],
+      type_ignores=[])
+
+
 Note that the only scenario in which this grammar change allows ``*Ts`` to be
 used as a direct annotation (rather than being wrapped in e.g. ``Tuple[*Ts]``)
 is ``*args``. Other uses are still invalid:
@@ -1045,14 +1080,18 @@ Implications
 As with the first grammar change, this change also has a number of side effects.
 In particular, the annotation of ``*args`` could be set to a starred object
 other than a ``TypeVarTuple`` - for example, the following nonsensical
-annotation is possible:
+annotations are possible:
 
 ::
 
-    >>> foo = [1, 2, 3]
-    >>> def bar(*args: *foo): pass  # Equivalent to `*args: tuple(foo)`
+    >>> foo = [1]
+    >>> def bar(*args: *foo): pass
     >>> bar.__annotations__
-    {'args': (1, 2, 3)}
+    {'args': 1}
+
+    >>> foo = [1, 2]
+    >>> def bar(*args: *foo): pass
+    ValueError: too many values to unpack (expected 1)
 
 Again, prevention of such annotations will need to be done by, say, static
 checkers, rather than at the level of syntax.