PEP 655: Integrate feedback from circa Feb 2021

[davidfstr]

This branch is easiest to review either:

1. a commit at a time, or
2. as a sequence of commits together but excluding the last commit "Eliminate many uses of '...' to decrease typographic noise".

CC sponsor @gvanrossum

[JelleZijlstra]

Welcome back!

[JelleZijlstra]

fyi, the typing-extensions implementation doesn't do this. Implementing this behavior would require the TypedDict constructor to inspect the annotations on each value, which is somewhat fragile and won't work if you use typing.TypedDict with typing_extensions.Required.

As the author of a tool that needs to understand this at runtime, I'd be OK with changing the spec so that Required and NotRequired aren't reflected in required_keys and optional_keys.

[davidfstr]

Implementing this behavior would require the TypedDict constructor to inspect the annotations on each value

Aye.

won't work if you use typing.TypedDict with typing_extensions.Required.

typing_extensions.Required is identical to typing.Required at runtime if the latter is available:

# typing_extensions.py
if hasattr(typing, 'Required'):
    Required = typing.Required
    NotRequired = typing.NotRequired

is somewhat fragile

Doesn't seem to me like the code to do the inspection would be very complex: It just has to unwrap Annotated[] values until it hits (Not)Required[] or something else. And then toss out the (Not)Required[] wrapper before stuffing the result into the appropiate __*_keys__ attribute.

[JelleZijlstra]

typing_extensions.Required is identical to typing.Required at runtime if the latter is available:

But on 3.8 through 3.10 typing.TypedDict exists and doesn't know about typing_extensions.Required.

[davidfstr]

As the author of a tool that needs to understand this at runtime, I'd be OK with changing the spec so that Required and NotRequired aren't reflected in required_keys and optional_keys.

👍

[davidfstr]

But on 3.8 through 3.10 typing.TypedDict exists and doesn't know about typing_extensions.Required.

Good point: I probably can't update typing.TypedDict in 3.8-3.10 to support a feature from 3.11.

Looks like the strategy in the past for this type of issue is to make typing_extensions.TypedDict recognize all supported syntax in the latest accepted typing PEPs, with users of typing.TypedDict sometimes receiving incomplete functionality. For example, under that interpretation:

• typing.TypedDict in Python 3.8-3.10 would not understand typing_extensions.Required and would put something like typing_extensions.Required[T] incorrectly into MyTypingDict.__optional_keys__ rather than putting T into MyTypingDict.__required_keys__.
• typing_extensions.TypedDict in Python 3.8-3.10 would correctly understand typing_extensions.Required.
• typing_extensions.TypedDict in Python 3.11+ would alias typing.TypedDict, which would correctly recognize both typing.Required and typing_extensions.Required (because the latter would alias the former).

Does that sound reasonable @JelleZijlstra ?

(If so I should update the "How to Teach This" section to recommend use of typing_extensions.TypedDict over typing.TypedDict in Python <= 3.10 when used in conjunction with typing_extensions.Required. 🎗️ )

[JelleZijlstra]

That would work and it's fine with me.

I'd still be OK with just skipping this work completely. My experience is that runtime checks in typing.py are fragile (in case a later PEP changes what kind of things are accepted at runtime) and often don't do exactly what I'd want when consuming the annotations. For example, in pyanalyze I'd still want to parse Required/NotRequired myself for use cases where users use 3.8-3.10 typing.TypedDict. Therefore, I'd generally favor minimizing runtime work and just making sure whatever the user wrote is recoverable through runtime introspection.

[AA-Turner]

Minor nit (GH doesn't seem to allow leaving review comments on unchanged lines -- this is line 17)

- `PEP 589 <https://www.python.org/dev/peps/pep-0589/>`__ defines syntax
+ :pep:`589` defines syntax

A

[davidfstr]

- `PEP 589 <https://www.python.org/dev/peps/pep-0589/>`__ defines syntax
+ :pep:`589` d

Thanks.

@AA-Turner Happen to know how I can adjust links to specific PEP sections with specific text? For example:

• From:

`all potentially-missing keys <https://www.python.org/dev/peps/pep-0589/#totality>`__

• To: (doesn't work!)

:pep:`all potentially-missing keys <589#totality>`

• To: (does work, but lacks my custom text)

:pep:`589#totality`

[AA-Turner]

@davidfstr Currently you can't do any of those (I believe even the :pep:`589#totality` form won't work with the current pep2html based system). I do have a commit to fix this at 91d908c (#2209), but it's in a large PR that also updates all implicit PEP NNN references to the explicit :pep:`NNN` form.

I could cherry-pick just that referenced commit and propose it as a PR, which would allow the "best" form (:pep:`all potentially-missing keys <589#totality>` to work, if it would be helpful?

A

[davidfstr]

I could cherry-pick just that referenced commit and propose it as a PR, which would allow the "best" form (:pep:`all potentially-missing keys <589#totality>` to work, if it would be helpful?

@AA-Turner Seems like you're planning to update all references anyway, so I have no preference on timing.

For now I'll leave the more-complex PEP links in their original URL-based form. If you happen to get improved :pep: syntax support merged before this branch, I'll happily update to use the new form.

[JelleZijlstra]

Happy to merge this whenever you want me to. We can also wait if Guido has any feedback.

[AA-Turner]

@davidfstr #2251 was just merged, so you should now be able to use :pep:`all potentially-missing keys <589#totality>`!

A

[davidfstr]

you should now be able to use :pep:`all potentially-missing keys <589#totality>`!

Done. Did rebase branch to tip of main (with the new syntax) and applied syntax in new commit at end of branch.

[davidfstr]

Happy to merge this whenever you want me to. We can also wait if Guido has any feedback.

Looks to me like all of the formatting issues have been resolved. There are 2 more-major pieces of feedback to integrate, but it would probably be easier to do that in (1) another PR to the PEP and (2) another mailing-list post to typing-sig. So @JelleZijlstra I'd advocate just merging this branch now that we're happy with it.

[AA-Turner]

I'm not Jelle, but I'll merge as I was on this PR anyway & Jelle already approved.