Moving typing_extensions stubs to typing_extensions

[srittau]

This is not an easy problem, and I don't expect that we will fix this soon. But I'd like to at least start a discussion about ways to move the typing_extensions stubs over to the typing_extensions repository. The advantages are the usual advantages for having stubs:

• Maintaining stubs and implementation at the same place is more likely to keep the stubs correct. (Although this is a smaller problem than usual due to the maintainer overlap.)
• This ensures that the installed version if typing_extensions matches the stubs.
• typing_extensions is not really part of the stdlib, although we pretend it is. While this is often useful, it can lead to problems. For example, users could use typing_extensions features without the module being installed, but type checkers and IDEs happily assuming it is.

This is a difficult problem, since we kind of need to assume that typing_extensions is installed. Even the stdlib stubs use it for newer typing features. I assume moving the stubs over wouldn't be much of a problem for Python-based type checkers like mypy, since they install the package anyway, but it could be a problem for non-Python based typing checkers.

One first step could be to add the stubs to the typing_extensions repository and ship them with the library (plus py.typed), but copy them over to _typeshed whenever they change. That way they would at least be maintained in the right place, and I don't think it would be extra work compared to now.

[JelleZijlstra]

Unfortunately I'm not sure this is feasible. The builtins stubs extensively use typing_extensions (from typing_extensions import Literal, SupportsIndex, TypeAlias, TypeGuard, final), so for mypy to work at all, you'll always need a version of typing_extensions with stubs installed on the system you are type checking. If you happen to have an old version of typing-extensions installed, mypy will likely crash horribly.

Relatedly, it means if we use typing-extensions features in typeshed, we're breaking compatibility with users who have older versions of typing-extensions installed locally.

[srittau]

Yes, the interdependencies between the stdlib and typing_extensions are one of the biggest problems. What do you think of maintaining the stubs at the typing_extensions repository at least, and copying them over to typeshed? I don't think this would require extra steps compared to now, but might make it a bit easier to keep them in sync.

[JelleZijlstra]

I'm not sure what we'd gain from that. Keeping things in two places introduces a risk that they'll go out of sync, and people who contribute to typing-extensions don't have much trouble contributing to typeshed too.

[hauntsaninja]

Strong -1 on this.

Note that we can't automatically guarantee typing_extensions is installed at all when users specify --python-executable (which is a very useful feature). I've also seen some real world code that assumes type checkers know what typing_extensions is (ie they do their imports under if TYPE_CHECKING and don't declare a dependency).

[AlexWaygood]

I absolutely agree that, in general, it's good to keep stubs close to the implementation. But I'm with @JelleZijlstra and @hauntsaninja on this — typing_extensions just seems too special to me. Any change in this area feels like it would cause a lot of pain for not much gain.

What do you think of maintaining the stubs at the typing_extensions repository at least, and copying them over to typeshed?

This seems like it would be viable but, like Jelle, I don't really see the benefit of doing this. It feels like it would just be extra busywork to have to maintain identical stubs in two separate repos :/

[AlexWaygood]

I think the consensus is that we shouldn't do this.