Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

PEP 655: Integrate feedback from circa Feb 2021 #2248

Merged
merged 15 commits into from
Jan 20, 2022
Merged

PEP 655: Integrate feedback from circa Feb 2021 #2248

Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
84897b2
Remove "Requires: 604" because that PEP has been released already
davidfstr Jan 2, 2022
52a57a5
Permit use of NotRequired[X|None] as a string annotation for Python 3…
davidfstr Jan 2, 2022
8ad4900
null -> None
davidfstr Jan 2, 2022
144caf5
Reference Implementation: Link to mypy and pyright implementations
davidfstr Jan 2, 2022
a1621ff
Add section: Interaction with Annotated[...]
davidfstr Jan 3, 2022
5c09b08
Add section: Interaction with __required_keys__ and __optional_keys__
davidfstr Jan 3, 2022
8f56383
Rejected Ideas: Special syntax around the *key* of a TypedDict item: …
davidfstr Jan 3, 2022
03ab8bc
Add section: Interaction with get_type_hints()
davidfstr Jan 17, 2022
59c5fcd
Add section: Interaction with get_origin() and get_args()
davidfstr Jan 17, 2022
d009d90
Clarify: The alternative syntax for TypedDict also supports (Not)Requ…
davidfstr Jan 17, 2022
a32f1f1
Mention existing meanings for +T, -T, ~T.
davidfstr Jan 17, 2022
f0a0787
Update Post-History
davidfstr Jan 17, 2022
9ae8574
Eliminate many uses of '...' to decrease typographic noise
davidfstr Jan 17, 2022
b1ea366
Use native PEP link
davidfstr Jan 19, 2022
3897926
Use improved PEP links from #2251
davidfstr Jan 20, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
182 changes: 144 additions & 38 deletions pep-0655.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,24 +6,22 @@ Discussions-To: typing-sig at python.org
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Requires: 604
Created: 30-Jan-2021
Python-Version: 3.11
Post-History: 31-Jan-2021, 11-Feb-2021, 20-Feb-2021, 26-Feb-2021
Post-History: 31-Jan-2021, 11-Feb-2021, 20-Feb-2021, 26-Feb-2021, 17-Jan-2022


Abstract
========

`PEP 589 <https://www.python.org/dev/peps/pep-0589/>`__ defines syntax
:pep:`589` defines syntax
for declaring a TypedDict with all required keys and syntax for defining
a TypedDict with `all potentially-missing
keys <https://www.python.org/dev/peps/pep-0589/#totality>`__ however it
a TypedDict with :pep:`all potentially-missing keys <589#totality>` however it
does not provide any syntax to declare some keys as required and others
as potentially-missing. This PEP introduces two new syntaxes:
``Required[...]`` which can be used on individual items of a
``Required[]`` which can be used on individual items of a
TypedDict to mark them as required, and
``NotRequired[...]`` which can be used on individual items
``NotRequired[]`` which can be used on individual items
to mark them as potentially-missing.


Expand Down Expand Up @@ -63,7 +61,7 @@ customary in other languages like TypeScript:
}

The difficulty is that the best word for marking a potentially-missing
key, ``Optional[...]``, is already used in Python for a completely
key, ``Optional[]``, is already used in Python for a completely
different purpose: marking values that could be either of a particular
type or ``None``. In particular the following does not work:

Expand All @@ -74,17 +72,17 @@ type or ``None``. In particular the following does not work:
year: Optional[int] # means int|None, not potentially-missing!

Attempting to use any synonym of “optional” to mark potentially-missing
keys (like ``Missing[...]``) would be too similar to ``Optional[...]``
keys (like ``Missing[]``) would be too similar to ``Optional[]``
and be easy to confuse with it.

Thus it was decided to focus on positive-form phrasing for required keys
instead, which is straightforward to spell as ``Required[...]``.
instead, which is straightforward to spell as ``Required[]``.

Nevertheless it is common for folks wanting to extend a regular
(``total=True``) TypedDict to only want to add a small number of
potentially-missing keys, which necessitates a way to mark keys that are
*not* required and potentially-missing, and so we also allow the
``NotRequired[...]`` form for that case.
``NotRequired[]`` form for that case.


Specification
Expand All @@ -109,10 +107,10 @@ potentially-missing key:
title: str
year: NotRequired[int]

It is an error to use ``Required[...]`` or ``NotRequired[...]`` in any
It is an error to use ``Required[]`` or ``NotRequired[]`` in any
location that is not an item of a TypedDict.

It is valid to use ``Required[...]`` and ``NotRequired[...]`` even for
It is valid to use ``Required[]`` and ``NotRequired[]`` even for
items where it is redundant, to enable additional explicitness if desired:

::
Expand All @@ -121,6 +119,96 @@ items where it is redundant, to enable additional explicitness if desired:
title: Required[str] # redundant
year: NotRequired[int]

It is an error to use both ``Required[]`` and ``NotRequired[]`` at the
same time:

::

class Movie(TypedDict):
title: str
year: NotRequired[Required[int]] # ERROR


The :pep:`alternative syntax <589#alternative-syntax>`
for TypedDict also supports
``Required[]`` and ``NotRequired[]``:

::

Movie = TypedDict('Movie', {'name': str, 'year': NotRequired[int]})


Interaction with ``Annotated[]``
-----------------------------------

``Required[]`` and ``NotRequired[]`` can be used with ``Annotated[]``,
in any nesting order:

::

class Movie(TypedDict):
title: str
year: NotRequired[Annotated[int, ValueRange(-9999, 9999)]] # ok

::

class Movie(TypedDict):
title: str
year: Annotated[NotRequired[int], ValueRange(-9999, 9999)] # ok


Interaction with ``get_type_hints()``
-------------------------------------

``typing.get_type_hints(...)`` applied to a TypedDict will by default
strip out any ``Required[]`` or ``NotRequired[]`` type qualifiers,
since these qualifiers are expected to be inconvenient for code
casually introspecting type annotations.

``typing.get_type_hints(..., include_extras=True)`` however
*will* retain ``Required[]`` and ``NotRequired[]`` type qualifiers,
for advanced code introspecting type annotations that
wishes to preserve *all* annotations in the original source:

::

class Movie(TypedDict):
title: str
year: NotRequired[int]

assert get_type_hints(Movie) == \
{'title': str, 'year': int}
assert get_type_hints(Movie, include_extras=True) == \
{'title': str, 'year': NotRequired[int]}


Interaction with ``get_origin()`` and ``get_args()``
----------------------------------------------------

``typing.get_origin()`` and ``typing.get_args()`` will be updated to
recognize ``Required[]`` and ``NotRequired[]``:

::

assert get_origin(Required[int]) is Required
assert get_args(Required[int]) == (int,)

assert get_origin(NotRequired[int]) is NotRequired
assert get_args(NotRequired[int]) == (int,)


Interaction with ``__required_keys__`` and ``__optional_keys__``
----------------------------------------------------------------

An item marked with ``Required[]`` will always appear
in the ``__required_keys__`` for its enclosing TypedDict. Similarly an item
marked with ``NotRequired[]`` will always appear in ``__optional_keys__``.

::

assert Movie.__required_keys__ == frozenset({'title'})
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fyi, the typing-extensions implementation doesn't do this. Implementing this behavior would require the TypedDict constructor to inspect the annotations on each value, which is somewhat fragile and won't work if you use typing.TypedDict with typing_extensions.Required.

As the author of a tool that needs to understand this at runtime, I'd be OK with changing the spec so that Required and NotRequired aren't reflected in required_keys and optional_keys.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Implementing this behavior would require the TypedDict constructor to inspect the annotations on each value

Aye.

won't work if you use typing.TypedDict with typing_extensions.Required.

typing_extensions.Required is identical to typing.Required at runtime if the latter is available:

# typing_extensions.py
if hasattr(typing, 'Required'):
    Required = typing.Required
    NotRequired = typing.NotRequired

is somewhat fragile

Doesn't seem to me like the code to do the inspection would be very complex: It just has to unwrap Annotated[] values until it hits (Not)Required[] or something else. And then toss out the (Not)Required[] wrapper before stuffing the result into the appropiate __*_keys__ attribute.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typing_extensions.Required is identical to typing.Required at runtime if the latter is available:

But on 3.8 through 3.10 typing.TypedDict exists and doesn't know about typing_extensions.Required.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As the author of a tool that needs to understand this at runtime, I'd be OK with changing the spec so that Required and NotRequired aren't reflected in required_keys and optional_keys.

👍

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But on 3.8 through 3.10 typing.TypedDict exists and doesn't know about typing_extensions.Required.

Good point: I probably can't update typing.TypedDict in 3.8-3.10 to support a feature from 3.11.

Looks like the strategy in the past for this type of issue is to make typing_extensions.TypedDict recognize all supported syntax in the latest accepted typing PEPs, with users of typing.TypedDict sometimes receiving incomplete functionality. For example, under that interpretation:

  • typing.TypedDict in Python 3.8-3.10 would not understand typing_extensions.Required and would put something like typing_extensions.Required[T] incorrectly into MyTypingDict.__optional_keys__ rather than putting T into MyTypingDict.__required_keys__.
  • typing_extensions.TypedDict in Python 3.8-3.10 would correctly understand typing_extensions.Required.
  • typing_extensions.TypedDict in Python 3.11+ would alias typing.TypedDict, which would correctly recognize both typing.Required and typing_extensions.Required (because the latter would alias the former).

Does that sound reasonable @JelleZijlstra ?

(If so I should update the "How to Teach This" section to recommend use of typing_extensions.TypedDict over typing.TypedDict in Python <= 3.10 when used in conjunction with typing_extensions.Required. 🎗️ )

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That would work and it's fine with me.

I'd still be OK with just skipping this work completely. My experience is that runtime checks in typing.py are fragile (in case a later PEP changes what kind of things are accepted at runtime) and often don't do exactly what I'd want when consuming the annotations. For example, in pyanalyze I'd still want to parse Required/NotRequired myself for use cases where users use 3.8-3.10 typing.TypedDict. Therefore, I'd generally favor minimizing runtime work and just making sure whatever the user wrote is recoverable through runtime introspection.

assert Movie.__optional_keys__ == frozenset({'year'})


Backwards Compatibility
=======================
Expand All @@ -133,16 +221,16 @@ How to Teach This

To define a TypedDict where most keys are required and some are
potentially-missing, define a single TypedDict as normal
and mark those few keys that are potentially-missing with ``NotRequired[...]``.
and mark those few keys that are potentially-missing with ``NotRequired[]``.

To define a TypedDict where most keys are potentially-missing and a few are
required, define a ``total=False`` TypedDict
and mark those few keys that are required with ``Required[...]``.
and mark those few keys that are required with ``Required[]``.

If some items accept ``None`` in addition to a regular value, it is
recommended that the ``TYPE|None`` syntax be preferred over
``Optional[TYPE]`` for marking such item values, to avoid using
``Required[...]`` or ``NotRequired[...]`` alongside ``Optional[...]``
``Required[]`` or ``NotRequired[]`` alongside ``Optional[]``
within the same TypedDict definition:

Yes:
Expand All @@ -155,7 +243,15 @@ Yes:
name: str
owner: NotRequired[str|None]

Avoid (unless Python 3.5-3.6):
Okay (required for Python 3.5.3-3.6):

::

class Dog(TypedDict):
name: str
owner: 'NotRequired[str|None]'

No:

::

Expand All @@ -168,15 +264,15 @@ Avoid (unless Python 3.5-3.6):
Reference Implementation
========================

The goal is to be able to make the following statement:

The `mypy <http://www.mypy-lang.org/>`__ type checker supports
``Required`` and ``NotRequired``. A reference implementation of the
runtime component is provided in the
`typing_extensions <https://github.com/python/typing/tree/master/typing_extensions>`__
module.
The `mypy <http://www.mypy-lang.org/>`__
`0.930 <https://mypy-lang.blogspot.com/2021/12/mypy-0930-released.html>`__
and `pyright <https://github.com/Microsoft/pyright>`__
`1.1.117 <https://github.com/microsoft/pyright/commit/7ed245b1845173090c6404e49912e8cbfb3417c8>`__
type checkers support ``Required`` and ``NotRequired``.

The mypy implementation is currently still being worked on.
A reference implementation of the runtime component is provided in the
`typing_extensions <https://github.com/python/typing/tree/master/typing_extensions>`__
module.


Rejected Ideas
Expand All @@ -189,19 +285,20 @@ Special syntax around the *key* of a TypedDict item

class MyThing(TypedDict):
opt1?: str # may not exist, but if exists, value is string
opt2: Optional[str] # always exists, but may have null value
opt2: Optional[str] # always exists, but may have None value

or:
This syntax would require Python grammar changes and it is not
believed that marking TypedDict items as required or potentially-missing
would meet the high bar required to make such grammar changes.

::

class MyThing(TypedDict):
Optional[opt1]: str # may not exist, but if exists, value is string
opt2: Optional[str] # always exists, but may have null value
opt2: Optional[str] # always exists, but may have None value

These syntaxes would require Python grammar changes and it is not
believed that marking TypedDict items as required or potentially-missing
would meet the high bar required to make such grammar changes.
This syntax causes ``Optional[]`` to take on different meanings depending
on where it is positioned, which is inconsistent and confusing.

Also, “let’s just not put funny syntax before the colon.” [1]_

Expand All @@ -216,13 +313,13 @@ with opposite-of-normal totality:
::

class MyThing(TypedDict, total=False):
req1: +int # + means a required key, or Required[...]
req1: +int # + means a required key, or Required[]
opt1: str
req2: +float

class MyThing(TypedDict):
req1: int
opt1: -str # - means a potentially-missing key, or NotRequired[...]
opt1: -str # - means a potentially-missing key, or NotRequired[]
req2: float

class MyThing(TypedDict):
Expand All @@ -235,10 +332,20 @@ Such operators could be implemented on ``type`` via the ``__pos__``,
grammar.

It was decided that it would be prudent to introduce longform syntax
(i.e. ``Required[...]`` and ``NotRequired[...]``) before introducing
(i.e. ``Required[]`` and ``NotRequired[]``) before introducing
any shortform syntax. Future PEPs may reconsider introducing this
or other shortform syntax options.

Note when reconsidering introducing this shortform syntax that
``+``, ``-``, and ``~`` already have existing meanings in the Python
typing world: covariant, contravariant, and invariant:

::

>>> from typing import TypeVar
>>> (TypeVar('T', covariant=True), TypeVar('U', contravariant=True), TypeVar('V'))
(+T, -U, ~V)


Marking absence of a value with a special constant
--------------------------------------------------
Expand Down Expand Up @@ -379,9 +486,8 @@ distinguishing between its analogous constants ``null`` and
Replace Optional with Nullable. Repurpose Optional to mean “optional item”.
---------------------------------------------------------------------------

``Optional[...]`` is too ubiquitous to deprecate. Although use of it
*may* fade over time in favor of the ``T|None`` syntax specified by `PEP
604 <https://www.python.org/dev/peps/pep-0604/>`__.
``Optional[]`` is too ubiquitous to deprecate. Although use of it
*may* fade over time in favor of the ``T|None`` syntax specified by :pep:`604`.


Change Optional to mean “optional item” in certain contexts instead of “nullable”
Expand All @@ -406,7 +512,7 @@ or:
opt1: Optional[str]

This would add more confusion for users because it would mean that in
*some* contexts the meaning of ``Optional[...]`` is different than in
*some* contexts the meaning of ``Optional[]`` is different than in
other contexts, and it would be easy to overlook the flag.


Expand Down