PEP 655: Integrate feedback from circa Feb 2021 (#2248)

pep-0655.rst

@@ -6,24 +6,22 @@ Discussions-To: typing-sig at python.org
 Status: Draft
 Type: Standards Track
 Content-Type: text/x-rst
-Requires: 604
 Created: 30-Jan-2021
 Python-Version: 3.11
-Post-History: 31-Jan-2021, 11-Feb-2021, 20-Feb-2021, 26-Feb-2021
+Post-History: 31-Jan-2021, 11-Feb-2021, 20-Feb-2021, 26-Feb-2021, 17-Jan-2022
 
 
 Abstract
 ========
 
-`PEP 589 <https://www.python.org/dev/peps/pep-0589/>`__ defines syntax
+:pep:`589` defines syntax
 for declaring a TypedDict with all required keys and syntax for defining
-a TypedDict with `all potentially-missing
-keys <https://www.python.org/dev/peps/pep-0589/#totality>`__ however it
+a TypedDict with :pep:`all potentially-missing keys <589#totality>` however it
 does not provide any syntax to declare some keys as required and others
 as potentially-missing. This PEP introduces two new syntaxes:
-``Required[...]`` which can be used on individual items of a
+``Required[]`` which can be used on individual items of a
 TypedDict to mark them as required, and
-``NotRequired[...]`` which can be used on individual items
+``NotRequired[]`` which can be used on individual items
 to mark them as potentially-missing.
 
 
@@ -63,7 +61,7 @@ customary in other languages like TypeScript:
    }
 
 The difficulty is that the best word for marking a potentially-missing
-key, ``Optional[...]``, is already used in Python for a completely
+key, ``Optional[]``, is already used in Python for a completely
 different purpose: marking values that could be either of a particular
 type or ``None``. In particular the following does not work:
 
@@ -74,17 +72,17 @@ type or ``None``. In particular the following does not work:
        year: Optional[int]  # means int|None, not potentially-missing!
 
 Attempting to use any synonym of “optional” to mark potentially-missing
-keys (like ``Missing[...]``) would be too similar to ``Optional[...]``
+keys (like ``Missing[]``) would be too similar to ``Optional[]``
 and be easy to confuse with it.
 
 Thus it was decided to focus on positive-form phrasing for required keys
-instead, which is straightforward to spell as ``Required[...]``.
+instead, which is straightforward to spell as ``Required[]``.
 
 Nevertheless it is common for folks wanting to extend a regular
 (``total=True``) TypedDict to only want to add a small number of
 potentially-missing keys, which necessitates a way to mark keys that are
 *not* required and potentially-missing, and so we also allow the
-``NotRequired[...]`` form for that case.
+``NotRequired[]`` form for that case.
 
 
 Specification
@@ -109,10 +107,10 @@ potentially-missing key:
        title: str
        year: NotRequired[int]
 
-It is an error to use ``Required[...]`` or ``NotRequired[...]`` in any
+It is an error to use ``Required[]`` or ``NotRequired[]`` in any
 location that is not an item of a TypedDict.
 
-It is valid to use ``Required[...]`` and ``NotRequired[...]`` even for
+It is valid to use ``Required[]`` and ``NotRequired[]`` even for
 items where it is redundant, to enable additional explicitness if desired:
 
 ::
@@ -121,6 +119,96 @@ items where it is redundant, to enable additional explicitness if desired:
        title: Required[str]  # redundant
        year: NotRequired[int]
 
+It is an error to use both ``Required[]`` and ``NotRequired[]`` at the
+same time:
+
+::
+
+   class Movie(TypedDict):
+       title: str
+       year: NotRequired[Required[int]]  # ERROR
+
+
+The :pep:`alternative syntax <589#alternative-syntax>`
+for TypedDict also supports
+``Required[]`` and ``NotRequired[]``:
+
+::
+
+   Movie = TypedDict('Movie', {'name': str, 'year': NotRequired[int]})
+
+
+Interaction with ``Annotated[]``
+-----------------------------------
+
+``Required[]`` and ``NotRequired[]`` can be used with ``Annotated[]``,
+in any nesting order:
+
+::
+
+   class Movie(TypedDict):
+       title: str
+       year: NotRequired[Annotated[int, ValueRange(-9999, 9999)]]  # ok
+
+::
+
+   class Movie(TypedDict):
+       title: str
+       year: Annotated[NotRequired[int], ValueRange(-9999, 9999)]  # ok
+
+
+Interaction with ``get_type_hints()``
+-------------------------------------
+
+``typing.get_type_hints(...)`` applied to a TypedDict will by default
+strip out any ``Required[]`` or ``NotRequired[]`` type qualifiers,
+since these qualifiers are expected to be inconvenient for code
+casually introspecting type annotations.
+
+``typing.get_type_hints(..., include_extras=True)`` however
+*will* retain ``Required[]`` and ``NotRequired[]`` type qualifiers,
+for advanced code introspecting type annotations that
+wishes to preserve *all* annotations in the original source:
+
+::
+
+   class Movie(TypedDict):
+       title: str
+       year: NotRequired[int]
+   
+   assert get_type_hints(Movie) == \
+       {'title': str, 'year': int}
+   assert get_type_hints(Movie, include_extras=True) == \
+       {'title': str, 'year': NotRequired[int]}
+
+
+Interaction with ``get_origin()`` and ``get_args()``
+----------------------------------------------------
+
+``typing.get_origin()`` and ``typing.get_args()`` will be updated to
+recognize ``Required[]`` and ``NotRequired[]``:
+
+::
+
+   assert get_origin(Required[int]) is Required
+   assert get_args(Required[int]) == (int,)
+   
+   assert get_origin(NotRequired[int]) is NotRequired
+   assert get_args(NotRequired[int]) == (int,)
+
+
+Interaction with ``__required_keys__`` and ``__optional_keys__``
+----------------------------------------------------------------
+
+An item marked with ``Required[]`` will always appear
+in the ``__required_keys__`` for its enclosing TypedDict. Similarly an item
+marked with ``NotRequired[]`` will always appear in ``__optional_keys__``.
+
+::
+
+   assert Movie.__required_keys__ == frozenset({'title'})
+   assert Movie.__optional_keys__ == frozenset({'year'})
+
 
 Backwards Compatibility
 =======================
@@ -133,16 +221,16 @@ How to Teach This
 
 To define a TypedDict where most keys are required and some are
 potentially-missing, define a single TypedDict as normal
-and mark those few keys that are potentially-missing with ``NotRequired[...]``.
+and mark those few keys that are potentially-missing with ``NotRequired[]``.
 
 To define a TypedDict where most keys are potentially-missing and a few are
 required, define a ``total=False`` TypedDict
-and mark those few keys that are required with ``Required[...]``.
+and mark those few keys that are required with ``Required[]``.
 
 If some items accept ``None`` in addition to a regular value, it is
 recommended that the ``TYPE|None`` syntax be preferred over
 ``Optional[TYPE]`` for marking such item values, to avoid using
-``Required[...]`` or ``NotRequired[...]`` alongside ``Optional[...]``
+``Required[]`` or ``NotRequired[]`` alongside ``Optional[]``
 within the same TypedDict definition:
 
 Yes:
@@ -155,7 +243,15 @@ Yes:
        name: str
        owner: NotRequired[str|None]
 
-Avoid (unless Python 3.5-3.6):
+Okay (required for Python 3.5.3-3.6):
+
+::
+
+   class Dog(TypedDict):
+       name: str
+       owner: 'NotRequired[str|None]'
+
+No:
 
 ::
 
@@ -168,15 +264,15 @@ Avoid (unless Python 3.5-3.6):
 Reference Implementation
 ========================
 
-The goal is to be able to make the following statement:
-
-    The `mypy <http://www.mypy-lang.org/>`__ type checker supports
-    ``Required`` and ``NotRequired``. A reference implementation of the
-    runtime component is provided in the
-    `typing_extensions <https://github.com/python/typing/tree/master/typing_extensions>`__
-    module.
+The `mypy <http://www.mypy-lang.org/>`__
+`0.930 <https://mypy-lang.blogspot.com/2021/12/mypy-0930-released.html>`__
+and `pyright <https://github.com/Microsoft/pyright>`__
+`1.1.117 <https://github.com/microsoft/pyright/commit/7ed245b1845173090c6404e49912e8cbfb3417c8>`__
+type checkers support ``Required`` and ``NotRequired``.
 
-The mypy implementation is currently still being worked on.
+A reference implementation of the runtime component is provided in the
+`typing_extensions <https://github.com/python/typing/tree/master/typing_extensions>`__
+module.
 
 
 Rejected Ideas
@@ -189,19 +285,20 @@ Special syntax around the *key* of a TypedDict item
 
    class MyThing(TypedDict):
        opt1?: str  # may not exist, but if exists, value is string
-       opt2: Optional[str]  # always exists, but may have null value
+       opt2: Optional[str]  # always exists, but may have None value
 
-or:
+This syntax would require Python grammar changes and it is not
+believed that marking TypedDict items as required or potentially-missing
+would meet the high bar required to make such grammar changes.
 
 ::
 
    class MyThing(TypedDict):
        Optional[opt1]: str  # may not exist, but if exists, value is string
-       opt2: Optional[str]  # always exists, but may have null value
+       opt2: Optional[str]  # always exists, but may have None value
 
-These syntaxes would require Python grammar changes and it is not
-believed that marking TypedDict items as required or potentially-missing
-would meet the high bar required to make such grammar changes.
+This syntax causes ``Optional[]`` to take on different meanings depending
+on where it is positioned, which is inconsistent and confusing.
 
 Also, “let’s just not put funny syntax before the colon.” [1]_
 
@@ -216,13 +313,13 @@ with opposite-of-normal totality:
 ::
 
    class MyThing(TypedDict, total=False):
-       req1: +int    # + means a required key, or Required[...]
+       req1: +int    # + means a required key, or Required[]
        opt1: str
        req2: +float
 
    class MyThing(TypedDict):
        req1: int
-       opt1: -str    # - means a potentially-missing key, or NotRequired[...]
+       opt1: -str    # - means a potentially-missing key, or NotRequired[]
        req2: float
 
    class MyThing(TypedDict):
@@ -235,10 +332,20 @@ Such operators could be implemented on ``type`` via the ``__pos__``,
 grammar.
 
 It was decided that it would be prudent to introduce longform syntax
-(i.e. ``Required[...]`` and ``NotRequired[...]``) before introducing
+(i.e. ``Required[]`` and ``NotRequired[]``) before introducing
 any shortform syntax. Future PEPs may reconsider introducing this
 or other shortform syntax options.
 
+Note when reconsidering introducing this shortform syntax that
+``+``, ``-``, and ``~`` already have existing meanings in the Python
+typing world: covariant, contravariant, and invariant:
+
+::
+
+   >>> from typing import TypeVar
+   >>> (TypeVar('T', covariant=True), TypeVar('U', contravariant=True), TypeVar('V'))
+   (+T, -U, ~V)
+
 
 Marking absence of a value with a special constant
 --------------------------------------------------
@@ -379,9 +486,8 @@ distinguishing between its analogous constants ``null`` and
 Replace Optional with Nullable. Repurpose Optional to mean “optional item”.
 ---------------------------------------------------------------------------
 
-``Optional[...]`` is too ubiquitous to deprecate. Although use of it
-*may* fade over time in favor of the ``T|None`` syntax specified by `PEP
-604 <https://www.python.org/dev/peps/pep-0604/>`__.
+``Optional[]`` is too ubiquitous to deprecate. Although use of it
+*may* fade over time in favor of the ``T|None`` syntax specified by :pep:`604`.
 
 
 Change Optional to mean “optional item” in certain contexts instead of “nullable”
@@ -406,7 +512,7 @@ or:
        opt1: Optional[str]
 
 This would add more confusion for users because it would mean that in
-*some* contexts the meaning of ``Optional[...]`` is different than in
+*some* contexts the meaning of ``Optional[]`` is different than in
 other contexts, and it would be easy to overlook the flag.