diff --git a/pep-0655.rst b/pep-0655.rst
index 012f88817b0..c4e499eebf4 100644
--- a/pep-0655.rst
+++ b/pep-0655.rst
@@ -6,24 +6,22 @@ Discussions-To: typing-sig at python.org
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
-Requires: 604
Created: 30-Jan-2021
Python-Version: 3.11
-Post-History: 31-Jan-2021, 11-Feb-2021, 20-Feb-2021, 26-Feb-2021
+Post-History: 31-Jan-2021, 11-Feb-2021, 20-Feb-2021, 26-Feb-2021, 17-Jan-2022
Abstract
========
-`PEP 589 `__ defines syntax
+:pep:`589` defines syntax
for declaring a TypedDict with all required keys and syntax for defining
-a TypedDict with `all potentially-missing
-keys `__ however it
+a TypedDict with :pep:`all potentially-missing keys <589#totality>` however it
does not provide any syntax to declare some keys as required and others
as potentially-missing. This PEP introduces two new syntaxes:
-``Required[...]`` which can be used on individual items of a
+``Required[]`` which can be used on individual items of a
TypedDict to mark them as required, and
-``NotRequired[...]`` which can be used on individual items
+``NotRequired[]`` which can be used on individual items
to mark them as potentially-missing.
@@ -63,7 +61,7 @@ customary in other languages like TypeScript:
}
The difficulty is that the best word for marking a potentially-missing
-key, ``Optional[...]``, is already used in Python for a completely
+key, ``Optional[]``, is already used in Python for a completely
different purpose: marking values that could be either of a particular
type or ``None``. In particular the following does not work:
@@ -74,17 +72,17 @@ type or ``None``. In particular the following does not work:
year: Optional[int] # means int|None, not potentially-missing!
Attempting to use any synonym of “optional” to mark potentially-missing
-keys (like ``Missing[...]``) would be too similar to ``Optional[...]``
+keys (like ``Missing[]``) would be too similar to ``Optional[]``
and be easy to confuse with it.
Thus it was decided to focus on positive-form phrasing for required keys
-instead, which is straightforward to spell as ``Required[...]``.
+instead, which is straightforward to spell as ``Required[]``.
Nevertheless it is common for folks wanting to extend a regular
(``total=True``) TypedDict to only want to add a small number of
potentially-missing keys, which necessitates a way to mark keys that are
*not* required and potentially-missing, and so we also allow the
-``NotRequired[...]`` form for that case.
+``NotRequired[]`` form for that case.
Specification
@@ -109,10 +107,10 @@ potentially-missing key:
title: str
year: NotRequired[int]
-It is an error to use ``Required[...]`` or ``NotRequired[...]`` in any
+It is an error to use ``Required[]`` or ``NotRequired[]`` in any
location that is not an item of a TypedDict.
-It is valid to use ``Required[...]`` and ``NotRequired[...]`` even for
+It is valid to use ``Required[]`` and ``NotRequired[]`` even for
items where it is redundant, to enable additional explicitness if desired:
::
@@ -121,6 +119,96 @@ items where it is redundant, to enable additional explicitness if desired:
title: Required[str] # redundant
year: NotRequired[int]
+It is an error to use both ``Required[]`` and ``NotRequired[]`` at the
+same time:
+
+::
+
+ class Movie(TypedDict):
+ title: str
+ year: NotRequired[Required[int]] # ERROR
+
+
+The :pep:`alternative syntax <589#alternative-syntax>`
+for TypedDict also supports
+``Required[]`` and ``NotRequired[]``:
+
+::
+
+ Movie = TypedDict('Movie', {'name': str, 'year': NotRequired[int]})
+
+
+Interaction with ``Annotated[]``
+-----------------------------------
+
+``Required[]`` and ``NotRequired[]`` can be used with ``Annotated[]``,
+in any nesting order:
+
+::
+
+ class Movie(TypedDict):
+ title: str
+ year: NotRequired[Annotated[int, ValueRange(-9999, 9999)]] # ok
+
+::
+
+ class Movie(TypedDict):
+ title: str
+ year: Annotated[NotRequired[int], ValueRange(-9999, 9999)] # ok
+
+
+Interaction with ``get_type_hints()``
+-------------------------------------
+
+``typing.get_type_hints(...)`` applied to a TypedDict will by default
+strip out any ``Required[]`` or ``NotRequired[]`` type qualifiers,
+since these qualifiers are expected to be inconvenient for code
+casually introspecting type annotations.
+
+``typing.get_type_hints(..., include_extras=True)`` however
+*will* retain ``Required[]`` and ``NotRequired[]`` type qualifiers,
+for advanced code introspecting type annotations that
+wishes to preserve *all* annotations in the original source:
+
+::
+
+ class Movie(TypedDict):
+ title: str
+ year: NotRequired[int]
+
+ assert get_type_hints(Movie) == \
+ {'title': str, 'year': int}
+ assert get_type_hints(Movie, include_extras=True) == \
+ {'title': str, 'year': NotRequired[int]}
+
+
+Interaction with ``get_origin()`` and ``get_args()``
+----------------------------------------------------
+
+``typing.get_origin()`` and ``typing.get_args()`` will be updated to
+recognize ``Required[]`` and ``NotRequired[]``:
+
+::
+
+ assert get_origin(Required[int]) is Required
+ assert get_args(Required[int]) == (int,)
+
+ assert get_origin(NotRequired[int]) is NotRequired
+ assert get_args(NotRequired[int]) == (int,)
+
+
+Interaction with ``__required_keys__`` and ``__optional_keys__``
+----------------------------------------------------------------
+
+An item marked with ``Required[]`` will always appear
+in the ``__required_keys__`` for its enclosing TypedDict. Similarly an item
+marked with ``NotRequired[]`` will always appear in ``__optional_keys__``.
+
+::
+
+ assert Movie.__required_keys__ == frozenset({'title'})
+ assert Movie.__optional_keys__ == frozenset({'year'})
+
Backwards Compatibility
=======================
@@ -133,16 +221,16 @@ How to Teach This
To define a TypedDict where most keys are required and some are
potentially-missing, define a single TypedDict as normal
-and mark those few keys that are potentially-missing with ``NotRequired[...]``.
+and mark those few keys that are potentially-missing with ``NotRequired[]``.
To define a TypedDict where most keys are potentially-missing and a few are
required, define a ``total=False`` TypedDict
-and mark those few keys that are required with ``Required[...]``.
+and mark those few keys that are required with ``Required[]``.
If some items accept ``None`` in addition to a regular value, it is
recommended that the ``TYPE|None`` syntax be preferred over
``Optional[TYPE]`` for marking such item values, to avoid using
-``Required[...]`` or ``NotRequired[...]`` alongside ``Optional[...]``
+``Required[]`` or ``NotRequired[]`` alongside ``Optional[]``
within the same TypedDict definition:
Yes:
@@ -155,7 +243,15 @@ Yes:
name: str
owner: NotRequired[str|None]
-Avoid (unless Python 3.5-3.6):
+Okay (required for Python 3.5.3-3.6):
+
+::
+
+ class Dog(TypedDict):
+ name: str
+ owner: 'NotRequired[str|None]'
+
+No:
::
@@ -168,15 +264,15 @@ Avoid (unless Python 3.5-3.6):
Reference Implementation
========================
-The goal is to be able to make the following statement:
-
- The `mypy `__ type checker supports
- ``Required`` and ``NotRequired``. A reference implementation of the
- runtime component is provided in the
- `typing_extensions `__
- module.
+The `mypy `__
+`0.930 `__
+and `pyright `__
+`1.1.117 `__
+type checkers support ``Required`` and ``NotRequired``.
-The mypy implementation is currently still being worked on.
+A reference implementation of the runtime component is provided in the
+`typing_extensions `__
+module.
Rejected Ideas
@@ -189,19 +285,20 @@ Special syntax around the *key* of a TypedDict item
class MyThing(TypedDict):
opt1?: str # may not exist, but if exists, value is string
- opt2: Optional[str] # always exists, but may have null value
+ opt2: Optional[str] # always exists, but may have None value
-or:
+This syntax would require Python grammar changes and it is not
+believed that marking TypedDict items as required or potentially-missing
+would meet the high bar required to make such grammar changes.
::
class MyThing(TypedDict):
Optional[opt1]: str # may not exist, but if exists, value is string
- opt2: Optional[str] # always exists, but may have null value
+ opt2: Optional[str] # always exists, but may have None value
-These syntaxes would require Python grammar changes and it is not
-believed that marking TypedDict items as required or potentially-missing
-would meet the high bar required to make such grammar changes.
+This syntax causes ``Optional[]`` to take on different meanings depending
+on where it is positioned, which is inconsistent and confusing.
Also, “let’s just not put funny syntax before the colon.” [1]_
@@ -216,13 +313,13 @@ with opposite-of-normal totality:
::
class MyThing(TypedDict, total=False):
- req1: +int # + means a required key, or Required[...]
+ req1: +int # + means a required key, or Required[]
opt1: str
req2: +float
class MyThing(TypedDict):
req1: int
- opt1: -str # - means a potentially-missing key, or NotRequired[...]
+ opt1: -str # - means a potentially-missing key, or NotRequired[]
req2: float
class MyThing(TypedDict):
@@ -235,10 +332,20 @@ Such operators could be implemented on ``type`` via the ``__pos__``,
grammar.
It was decided that it would be prudent to introduce longform syntax
-(i.e. ``Required[...]`` and ``NotRequired[...]``) before introducing
+(i.e. ``Required[]`` and ``NotRequired[]``) before introducing
any shortform syntax. Future PEPs may reconsider introducing this
or other shortform syntax options.
+Note when reconsidering introducing this shortform syntax that
+``+``, ``-``, and ``~`` already have existing meanings in the Python
+typing world: covariant, contravariant, and invariant:
+
+::
+
+ >>> from typing import TypeVar
+ >>> (TypeVar('T', covariant=True), TypeVar('U', contravariant=True), TypeVar('V'))
+ (+T, -U, ~V)
+
Marking absence of a value with a special constant
--------------------------------------------------
@@ -379,9 +486,8 @@ distinguishing between its analogous constants ``null`` and
Replace Optional with Nullable. Repurpose Optional to mean “optional item”.
---------------------------------------------------------------------------
-``Optional[...]`` is too ubiquitous to deprecate. Although use of it
-*may* fade over time in favor of the ``T|None`` syntax specified by `PEP
-604 `__.
+``Optional[]`` is too ubiquitous to deprecate. Although use of it
+*may* fade over time in favor of the ``T|None`` syntax specified by :pep:`604`.
Change Optional to mean “optional item” in certain contexts instead of “nullable”
@@ -406,7 +512,7 @@ or:
opt1: Optional[str]
This would add more confusion for users because it would mean that in
-*some* contexts the meaning of ``Optional[...]`` is different than in
+*some* contexts the meaning of ``Optional[]`` is different than in
other contexts, and it would be easy to overlook the flag.