bpo-44863: In TypedDict allow inherit from Generic and preserve bases #27663

sransara · 2021-08-07T19:05:51Z

Copied from: https://bugs.python.org/issue44863

TypedDict PEP-589 says:
A TypedDict cannot inherit from both a TypedDict type and a non-TypedDict base class.

So the current implementation has:
if type(base) is not _TypedDictMeta: raise TypeError(...)

This restricts the user from defining generic TypedDicts in the natural class based syntax:
class Pager(TypedDict, Generic[T]): ...

Although PEP 589 doesn't explicitly state generic support, I believe it is complete in covering the specification even if generics were involved (at least for the class based syntax).

I have tried putting together a PEP from guidance of typing-sig https://github.com/sransara/py-generic-typeddict/blob/master/pep-9999.rst. There is not much new contributions by that draft, except for specifying the alternative syntax and being more explicit about Generics.

So I'm wondering if it would be possible to relax the constraint: TypedDict inheritance to include Generic. In my point of view Generic is more of a mixin, so it doesn't go against the PEP 589. Or is this change big enough to warrant a PEP?

https://bugs.python.org/issue44863

the-knights-who-say-ni · 2021-08-07T19:05:54Z

Hello, and thanks for your contribution!

I'm a bot set up to make sure that the project can legally accept this contribution by verifying everyone involved has signed the PSF contributor agreement (CLA).

Recognized GitHub username

We couldn't find a bugs.python.org (b.p.o) account corresponding to the following GitHub usernames:

@sransara

This might be simply due to a missing "GitHub Name" entry in one's b.p.o account settings. This is necessary for legal reasons before we can look at this contribution. Please follow the steps outlined in the CPython devguide to rectify this issue.

You can check yourself to see if the CLA has been received.

Thanks again for the contribution, we look forward to reviewing it!

Lib/typing.py

Lib/test/test_typing.py

uriyyo

Could you please add News file using blurb? You can find more information here.

Lib/typing.py

sransara · 2021-09-03T10:55:31Z

I have one issue that I'm not sure if needs to be resolved.

_GenericAlias doesn't proxy the dunder attributes. So we are not able to say:

class A(TypedDict, Generic[T]):
    foo: T

assert(A[str].__required_keys__  == frozenset(['foo']) # Raises Attribute error
assert(A[str].__origin__.__required_keys__  == frozenset(['foo']) # Works

I'm just mulling over if the __class_getitem__ should also be overridden just to override the __getattr__.

Any opinions or ideas?

Edit 1: In a following commit, I chose to override __getitem__ and make it return from types.GenericAlias, because that seems to do the __origin__ proxying as I wanted (I'm not entirely sure how types.GenericAlias does the job though).

uriyyo · 2021-09-03T13:19:58Z

Lib/typing.py

+        params = tuple(_type_check(p, msg) for p in params)
+        nparams = len(cls.__dict__.get("__parameters__", ()))
+        _check_generic(cls, params, nparams)
+        return types.GenericAlias(cls, params)


I think there are no need to define __getitem__ method as far as Generic class has __class_getitem__ method.

I had two issues by not overriding the default:

Dunder attributes are not proxied

class A(TypedDict, Generic[T]): foo: T assert(A[str].__required_keys__ == frozenset(['foo']) # Raises Attribute error assert(A[str].__origin__.__required_keys__ == frozenset(['foo']) # Works

May be a bug in the original code. But even without the changes in this PR, following is valid at runtime (digging in what I think happens is that it inherits a __getitem__ from dict meta):

class G(TypedDict): g: str print(G[int]) # G is subscriptable although not a Generic

@Fidget-Spinner What is your opinion about issue mentioned by Samodya?

For 1st issue:
That happens because _BaseGenericAlias.__getattr__ intentionally doesn't proxy dunders to self.__origin__ https://github.com/python/cpython/blob/main/Lib/typing.py#L966. I don't know what's the correct behavior until I read the final version of the proposed PEP update/ PEP. Sorry.

For the 2nd behavior:
Since python 3.9, builtin generics support subscripting thanks to PEP 585. E.g. dict[str, int]. Since subclassing TypedDict subclasses dict, it also pulls dict.__class_getitem__ (and hence the type supports arbitrary subscripting). A quick way to verify this:

>>> type(G[int]) <class 'types.GenericAlias'> >>> G.__mro__ (<class '__main__.G'>, <class 'dict'>, <class 'object'>)

Re 1: Dunder proxying behavior isn't specified in the PEP draft either :) I switched gears to add this feature through a bpo and implementation as I am not sure whether this change needed a PEP/PEP update (my reasoning in the ticket description). By any chance do you know a PEP that specifies the behavior of GenericAlias for another type?

Re 2: I see. Thank you. But that does seem like something that shouldn't be allowed for a non generic Typeddict (Not a big deal though since it doesn't really do anything).

Considering following is the behavior for normal Generic class, I will revert the overriding of __getitem__ and not proxy the dunder attributes.

class A(Generic[T]): a: T print(A.__annotations__) # {'a': ~T} print(A[str].__annotations__) # raises AttributeError print(A[str].__origin__.__annotations__) # {'a': ~T}

sobolevn

Awesome feature! I've been waiting for it for a long time 🎉

Lib/test/test_typing.py

Fidget-Spinner

I left some small notes below. I'll try to do a full and more thorough review eventually when the PEP solidifies. Thanks for moving this change forward!

Fidget-Spinner · 2021-09-05T07:50:21Z

Lib/typing.py

+        params = tuple(_type_check(p, msg) for p in params)
+        nparams = len(cls.__dict__.get("__parameters__", ()))
+        _check_generic(cls, params, nparams)
+        return types.GenericAlias(cls, params)


Is there a reason why you're using the builtin types.GenericAlias vs the typing typing._GenericAlias version?

IMO, I'd prefer you use the typing version. It's much easier to subclass and customize. It also supports older Python versions if you're porting this to typing_extensions. types.GenericAlias only works for Python 3.9 onwards. (Also, I think there isn't a precedent for things in typing returning a types.GenericAlias.)

types.GenericAlias was chosen just to circumvent the fact that typing._GenericAlias doesn't proxy the dunder attributes (specifically __required_keys__, __optional_keys__ etc.). Given your reasoning we can make the change to override __getattr__ of typing._GenericAlias. That is assuming we want to proxy those attributes.

Fidget-Spinner · 2021-09-05T08:11:30Z

Lib/typing.py

+        params = tuple(_type_check(p, msg) for p in params)
+        nparams = len(cls.__dict__.get("__parameters__", ()))
+        _check_generic(cls, params, nparams)
+        return types.GenericAlias(cls, params)


For 1st issue:
That happens because _BaseGenericAlias.__getattr__ intentionally doesn't proxy dunders to self.__origin__ https://github.com/python/cpython/blob/main/Lib/typing.py#L966. I don't know what's the correct behavior until I read the final version of the proposed PEP update/ PEP. Sorry.

For the 2nd behavior:
Since python 3.9, builtin generics support subscripting thanks to PEP 585. E.g. dict[str, int]. Since subclassing TypedDict subclasses dict, it also pulls dict.__class_getitem__ (and hence the type supports arbitrary subscripting). A quick way to verify this:

>>> type(G[int]) <class 'types.GenericAlias'> >>> G.__mro__ (<class '__main__.G'>, <class 'dict'>, <class 'object'>)

Misc/NEWS.d/next/Library/2021-09-03-07-56-48.bpo-44863.udgz95.rst

Co-authored-by: Ken Jin <28750310+Fidget-Spinner@users.noreply.github.com>

sransara · 2022-05-02T13:24:25Z

@serhiy-storchaka Agreed that this PR shouldn't try to meddle with previously allowed behavior in non generic TypedDicts. Using the suggestion from your comment, the commit 80e9104 reverts forbidding of parametrization of non-generic TypedDict types.

JelleZijlstra

We're missing one big thing: documentation. We should add a versionchanged: 3.11 note in https://docs.python.org/3.11/library/typing.html#typing.TypedDict saying that TypedDicts can now be generic.

JelleZijlstra · 2022-05-02T14:26:31Z

Lib/typing.py

@@ -1775,7 +1775,9 @@ def __init_subclass__(cls, *args, **kwargs):
        if '__orig_bases__' in cls.__dict__:
            error = Generic in cls.__orig_bases__
        else:
-            error = Generic in cls.__bases__ and cls.__name__ != 'Protocol'
+            error = (Generic in cls.__bases__ and
+                        cls.__name__ != 'Protocol' and


I see the existing code did this already, but checking the class name only seems wrong. Let me see if I can find a user-visible bug based on this.

It could be written as type(cls) != _ProtocolMeta, but let left this to other issue.

Lib/typing.py

serhiy-storchaka · 2022-05-02T17:57:41Z

Lib/typing.py

@@ -1775,7 +1775,9 @@ def __init_subclass__(cls, *args, **kwargs):
        if '__orig_bases__' in cls.__dict__:
            error = Generic in cls.__orig_bases__
        else:
-            error = Generic in cls.__bases__ and cls.__name__ != 'Protocol'
+            error = (Generic in cls.__bases__ and
+                        cls.__name__ != 'Protocol' and


It could be written as type(cls) != _ProtocolMeta, but let left this to other issue.

Lib/typing.py

JelleZijlstra · 2022-05-02T18:19:36Z

I'll update the branch to apply Serhyi's suggestions and add docs.

@serhiy-storchaka are you OK with merging this and the generic namedtuple PR once your comments are resolved?

…t-simple

serhiy-storchaka · 2022-05-02T18:51:54Z

Yes, I am OK with this. I just want to write tests that subscription works for arbitrary TypedDicts and named tuples (since the idea of forbidding it was rejected).

davidfstr · 2022-05-03T15:28:20Z

I just want to write tests that subscription works for arbitrary TypedDicts and named tuples (since the idea of forbidding it was rejected).

At least on my end I was rejecting subscription arbitrary TypedDicts in previous Python versions that happened to be forward-compatible with the use of generic TypeVars proposed in this thread.

I think it is questionable to allow subscripting TypedDicts in a way incompatible with this thread since then the semantics of that subscripting isn't clear.

JelleZijlstra · 2022-05-03T15:43:35Z

At least on my end I was rejecting subscription arbitrary TypedDicts in previous Python versions that happened to be forward-compatible with the use of generic TypeVars proposed in this thread.

Subscripting TypedDict classes worked in 3.9 and 3.10, but not 3.8. I assume that was accidental. But now that the cat is out of the bag, I'd rather not change the user-visible behavior.

theahura · 2022-07-18T17:40:06Z

@JelleZijlstra just to clarify, there are currently no plans to support generic typeddicts in 3.8? That would be a bummer

AlexWaygood · 2022-07-18T17:45:13Z

@JelleZijlstra just to clarify, there are currently no plans to support generic typeddicts in 3.8? That would be a bummer

Python 3.8 is in security-only mode (and has been for a while). This is a new feature, so being able to create generic typing.TypedDicts won't be backported; you'll only get it on Python 3.11+.

typing_extensions.TypedDict (the third-party backport) now supports generic TypedDicts, however. If you're using pyright, you should be able to use generic typing_extensions.TypedDicts on Python 3.8 right now. (Mypy doesn't support them yet; see python/mypy#3863. I don't know about other type-checkers.)

@AlexWaygood

Changes: # Release 4.3.0 (July 1, 2022) - Add `typing_extensions.NamedTuple`, allowing for generic `NamedTuple`s on Python <3.11 (backport from python/cpython#92027, by Serhiy Storchaka). Patch by Alex Waygood (@AlexWaygood). - Adjust `typing_extensions.TypedDict` to allow for generic `TypedDict`s on Python <3.11 (backport from python/cpython#27663, by Samodya Abey). Patch by Alex Waygood (@AlexWaygood). # Release 4.2.0 (April 17, 2022) - Re-export `typing.Unpack` and `typing.TypeVarTuple` on Python 3.11. - Add `ParamSpecArgs` and `ParamSpecKwargs` to `__all__`. - Improve "accepts only single type" error messages. - Improve the distributed package. Patch by Marc Mueller (@cdce8p). - Update `typing_extensions.dataclass_transform` to rename the `field_descriptors` parameter to `field_specifiers` and accept arbitrary keyword arguments. - Add `typing_extensions.get_overloads` and `typing_extensions.clear_overloads`, and add registry support to `typing_extensions.overload`. Backport from python/cpython#89263. - Add `typing_extensions.assert_type`. Backport from bpo-46480. - Drop support for Python 3.6. Original patch by Adam Turner (@AA-Turner). # Release 4.1.1 (February 13, 2022) - Fix importing `typing_extensions` on Python 3.7.0 and 3.7.1. Original patch by Nikita Sobolev (@sobolevn). # Release 4.1.0 (February 12, 2022) - Runtime support for PEP 646, adding `typing_extensions.TypeVarTuple` and `typing_extensions.Unpack`. - Add interaction of `Required` and `NotRequired` with `__required_keys__`, `__optional_keys__` and `get_type_hints()`. Patch by David Cabot (@d-k-bo). - Runtime support for PEP 675 and `typing_extensions.LiteralString`. - Add `Never` and `assert_never`. Backport from bpo-46475. - `ParamSpec` args and kwargs are now equal to themselves. Backport from bpo-46676. Patch by Gregory Beauregard (@GBeauregard). - Add `reveal_type`. Backport from bpo-46414. - Runtime support for PEP 681 and `typing_extensions.dataclass_transform`. - `Annotated` can now wrap `ClassVar` and `Final`. Backport from bpo-46491. Patch by Gregory Beauregard (@GBeauregard). - Add missed `Required` and `NotRequired` to `__all__`. Patch by Yuri Karabas (@uriyyo). - The `@final` decorator now sets the `__final__` attribute on the decorated object to allow runtime introspection. Backport from bpo-46342. - Add `is_typeddict`. Patch by Chris Moradi (@chrismoradi) and James Hilton-Balfe (@Gobot1234). # Release 4.0.1 (November 30, 2021) - Fix broken sdist in release 4.0.0. Patch by Adam Turner (@AA-Turner). - Fix equality comparison for `Required` and `NotRequired`. Patch by Jelle Zijlstra (@JelleZijlstra). - Fix usage of `Self` as a type argument. Patch by Chris Wesseling (@CharString) and James Hilton-Balfe (@Gobot1234). # Release 4.0.0 (November 14, 2021) - Starting with version 4.0.0, typing_extensions uses Semantic Versioning. See the README for more information. - Dropped support for Python versions 3.5 and older, including Python 2.7. - Simplified backports for Python 3.6.0 and newer. Patch by Adam Turner (@AA-Turner). ## Added in version 4.0.0 - Runtime support for PEP 673 and `typing_extensions.Self`. Patch by James Hilton-Balfe (@Gobot1234). - Runtime support for PEP 655 and `typing_extensions.Required` and `NotRequired`. Patch by David Foster (@davidfstr). ## Removed in version 4.0.0 The following non-exported but non-private names have been removed as they are unneeded for supporting Python 3.6 and newer. - TypingMeta - OLD_GENERICS - SUBS_TREE - HAVE_ANNOTATED - HAVE_PROTOCOLS - V_co - VT_co # Previous releases Prior to release 4.0.0 we did not provide a changelog. Please check the Git history for details.

@Gobot1234

…thon-3.10.8 - Updated from version 4.1.1 to 4.4.0 - Update of rootfile - Changelog # Release 4.4.0 (October 6, 2022) - Add `typing_extensions.Any` a backport of python 3.11's Any class which is subclassable at runtime. (backport from python/cpython#31841, by Shantanu and Jelle Zijlstra). Patch by James Hilton-Balfe (@Gobot1234). - Add initial support for TypeVarLike `default` parameter, PEP 696. Patch by Marc Mueller (@cdce8p). - Runtime support for PEP 698, adding `typing_extensions.override`. Patch by Jelle Zijlstra. - Add the `infer_variance` parameter to `TypeVar`, as specified in PEP 695. Patch by Jelle Zijlstra. # Release 4.3.0 (July 1, 2022) - Add `typing_extensions.NamedTuple`, allowing for generic `NamedTuple`s on Python <3.11 (backport from python/cpython#92027, by Serhiy Storchaka). Patch by Alex Waygood (@AlexWaygood). - Adjust `typing_extensions.TypedDict` to allow for generic `TypedDict`s on Python <3.11 (backport from python/cpython#27663, by Samodya Abey). Patch by Alex Waygood (@AlexWaygood). # Release 4.2.0 (April 17, 2022) - Re-export `typing.Unpack` and `typing.TypeVarTuple` on Python 3.11. - Add `ParamSpecArgs` and `ParamSpecKwargs` to `__all__`. - Improve "accepts only single type" error messages. - Improve the distributed package. Patch by Marc Mueller (@cdce8p). - Update `typing_extensions.dataclass_transform` to rename the `field_descriptors` parameter to `field_specifiers` and accept arbitrary keyword arguments. - Add `typing_extensions.get_overloads` and `typing_extensions.clear_overloads`, and add registry support to `typing_extensions.overload`. Backport from python/cpython#89263. - Add `typing_extensions.assert_type`. Backport from bpo-46480. - Drop support for Python 3.6. Original patch by Adam Turner (@AA-Turner). Tested-by: Adolf Belka <adolf.belka@ipfire.org> Signed-off-by: Adolf Belka <adolf.belka@ipfire.org>

sransara added 2 commits August 7, 2021 08:20

Allow TypedDict to inherit from Generic

6cfdc5f

TypedDict preserve MRO

c022e44

sransara requested a review from gvanrossum as a code owner August 7, 2021 19:05

the-knights-who-say-ni added the CLA not signed label Aug 7, 2021

bedevere-bot added the awaiting review label Aug 7, 2021

the-knights-who-say-ni added CLA signed and removed CLA not signed labels Aug 11, 2021

uriyyo reviewed Sep 1, 2021

View reviewed changes

Lib/typing.py Outdated Show resolved Hide resolved

uriyyo reviewed Sep 1, 2021

View reviewed changes

Lib/test/test_typing.py Outdated Show resolved Hide resolved

uriyyo reviewed Sep 1, 2021

View reviewed changes

Fix base classes to be just generic and dict

2fa00b8

sransara requested a review from Fidget-Spinner as a code owner September 3, 2021 02:03

uriyyo reviewed Sep 3, 2021

View reviewed changes

Lib/typing.py Outdated Show resolved Hide resolved

sransara added 4 commits September 3, 2021 06:50

Simplify addiotion of Generic base class

b829d8a

Fully proxy origin by using types.GenericAlias

c5762df

Add NEWS blurb

4c5db0a

Add better NEWS blurb

34189dd

uriyyo reviewed Sep 3, 2021

View reviewed changes

Add more testcases

0cfd637

sobolevn reviewed Sep 4, 2021

View reviewed changes

Lib/test/test_typing.py Show resolved Hide resolved

sransara added 2 commits September 4, 2021 08:16

Add implicit any test case

5e808af

Fix issue with plain generic inheritance

a2482ff

sransara requested a review from uriyyo September 4, 2021 15:01

Include bases assertion

071c1b6

Fidget-Spinner reviewed Sep 5, 2021

View reviewed changes

sransara and others added 2 commits September 5, 2021 03:34

Update NEWS blurb with better formatting

3ce517b

Co-authored-by: Ken Jin <28750310+Fidget-Spinner@users.noreply.github.com>

Revert overriding of getitem and not proxy dunders

2f003e0

Fidget-Spinner added the DO-NOT-MERGE label Sep 5, 2021

sransara requested review from JelleZijlstra and serhiy-storchaka May 2, 2022 13:27

Fix whitespacing with reindent.py

56b69e0

JelleZijlstra reviewed May 2, 2022

View reviewed changes

serhiy-storchaka reviewed May 2, 2022

View reviewed changes

JelleZijlstra added 3 commits May 2, 2022 12:20

fix tuple

4a40825

remove unnecessary __class_getitem__ override

4b50ae2

Merge remote-tracking branch 'upstream/main' into py-generic-typeddic…

ecc7726

…t-simple

docs

5b5a983

JelleZijlstra removed the DO-NOT-MERGE label May 2, 2022

Merge branch 'main' into py-generic-typeddict-simple

99a1430

JelleZijlstra self-assigned this May 2, 2022

Merge branch 'main' into py-generic-typeddict-simple

79c2bb4

JelleZijlstra merged commit f6f36cc into python:main May 3, 2022

bedevere-bot removed the awaiting review label May 3, 2022

This was referenced May 3, 2022

Allow TypedDict to inherit from Generics #89026

Closed

Backport generic NamedTuple and TypedDict support python/typing_extensions#7

Closed

erictraut mentioned this pull request May 3, 2022

Add support for generic TypedDict microsoft/pyright#3419

Closed

AlexWaygood mentioned this pull request May 27, 2022

Backport generic TypedDicts python/typing_extensions#46

Merged

not-my-profile mentioned this pull request Dec 28, 2022

Runtime TypeError caused by mixing typing and typing_extensions not caught microsoft/pyright#4380

Closed

rmartin16 mentioned this pull request Oct 14, 2023

Integrate typing into source rmartin16/qbittorrent-api#345

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bpo-44863: In TypedDict allow inherit from Generic and preserve bases #27663

bpo-44863: In TypedDict allow inherit from Generic and preserve bases #27663

sransara commented Aug 7, 2021 •

edited by bedevere-bot

Loading

the-knights-who-say-ni commented Aug 7, 2021

uriyyo left a comment

sransara commented Sep 3, 2021 •

edited

Loading

uriyyo Sep 3, 2021

sransara Sep 3, 2021 •

edited

Loading

uriyyo Sep 3, 2021

Fidget-Spinner Sep 5, 2021

sransara Sep 5, 2021

sransara Sep 5, 2021 •

edited

Loading

sobolevn left a comment

Fidget-Spinner left a comment

Fidget-Spinner Sep 5, 2021

sransara Sep 5, 2021

Fidget-Spinner Sep 5, 2021

sransara commented May 2, 2022

JelleZijlstra left a comment

JelleZijlstra May 2, 2022

serhiy-storchaka May 2, 2022

serhiy-storchaka May 2, 2022

JelleZijlstra commented May 2, 2022

serhiy-storchaka commented May 2, 2022

davidfstr commented May 3, 2022

JelleZijlstra commented May 3, 2022

theahura commented Jul 18, 2022

AlexWaygood commented Jul 18, 2022 •

edited

Loading

bpo-44863: In TypedDict allow inherit from Generic and preserve bases #27663

bpo-44863: In TypedDict allow inherit from Generic and preserve bases #27663

Conversation

sransara commented Aug 7, 2021 • edited by bedevere-bot Loading

the-knights-who-say-ni commented Aug 7, 2021

Recognized GitHub username

uriyyo left a comment

Choose a reason for hiding this comment

sransara commented Sep 3, 2021 • edited Loading

Choose a reason for hiding this comment

sransara Sep 3, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sransara Sep 5, 2021 • edited Loading

Choose a reason for hiding this comment

sobolevn left a comment

Choose a reason for hiding this comment

Fidget-Spinner left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sransara commented May 2, 2022

JelleZijlstra left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JelleZijlstra commented May 2, 2022

serhiy-storchaka commented May 2, 2022

davidfstr commented May 3, 2022

JelleZijlstra commented May 3, 2022

theahura commented Jul 18, 2022

AlexWaygood commented Jul 18, 2022 • edited Loading

sransara commented Aug 7, 2021 •

edited by bedevere-bot

Loading

sransara commented Sep 3, 2021 •

edited

Loading

sransara Sep 3, 2021 •

edited

Loading

sransara Sep 5, 2021 •

edited

Loading

AlexWaygood commented Jul 18, 2022 •

edited

Loading