Mixins with generic self and type bounds

[zero323]

Let's say I have a class

class Foo: ...

and a typevar with bound

from typing import TypeVar

T = TypeVar("T", bound=Foo)

Now I'd like to define a class that will be used as a mixin:

class MixIn:
    def foo(self: T) -> T: ...

I want to express two things here:

• MixIn should be used only as a mixin with Foo (the idea is similar to Scala self-type. In other words this should type check:

  class Bar(Foo, MixIn): ...

  while this shouldn't:

  class Baz(MixIn): ...

• MixIn.foo returns the same type as self (at least Foo with MixIn).

Additionally it should address issues like #5868 or #5837 with zero boilerplate.

Right now (0.720+dev.e479b6d55f927ed7b71d94e8632dc3921b983362) this fails:

from typing import TypeVar

class Foo: ...

T = TypeVar("T", bound=Foo)

class MixIn:
    def foo(self: T) -> T: ...

mypy --python-version 3.7 --strict-optional --no-site-packages --show-traceback --no-implicit-optional mixinwithgenericselfandbounds.py
mixinwithgenericselfandbounds.py:8: error: The erased type of self "mixinwithgenericselfandbounds.Foo" is not a supertype of its class "mixinwithgenericselfandbounds.MixIn"

[canassa]

I am having the same issue with the mixins I wrote for Django models, e.g.:

T = TypeVar("T", bound=models.Model)


class SafeCreateMixin:
    @classmethod
    @transaction.atomic()
    def safe_create(cls: Type[T], **kwargs) -> Optional[T]:
        try:
            return cls.objects.create(**kwargs)
        except IntegrityError as e:
            if not is_unique_violation(e):
                raise
            return None

It fails with the following error:

error: The erased type of self "Type[django.db.models.base.Model]" is not a supertype of its class "Type[core.utils.utils.SafeCreateMixin]"

[ilevkivskyi]

These kind of restrictions on self are not yet supported, but I think this can be a useful feature to support (there are also several other cases where restrictions and/or overloads on self are handy).

[ilevkivskyi]

See e.g. python/typeshed#3110 (comment) for other uses of explicit self restrictions.

[ziima]

@zero323: class Baz(MixIn): should pass the type check, since Baz may be just another mixin.

[zero323]

@zero323: class Baz(MixIn): should pass the type check, since Baz may be just another mixin.

@ziima Fair point, but I am not convinced that such transitive notion has any practical applications.

It would effectively mean that the type check on self should be applied only when Baz is instantiated. If not, such type check would be meaningless (further restrictions, like requiring abstract classes, notwithstanding), as you cannot prove, in the open-world, that Baz, or some class that mixes-in Baz, won't be mixed with Foo later in time.

Personally I'd say that a specific error, that can be selectively suppressed, is much more useful in such case, i.e.:

class Baz(MixIn): ...  # Error: Cannot prove that "Baz" is a subclass of "Foo" [self-bound]

I'd say that if you really want to type check such cases self bound should be explicit, i.e.

class Foz(MixIn):
    def foz(self: T) -> None: ...

but that might unnecessarily complicate the implementation.

[ziima]

Fair point, but I am not convinced that such transitive notion has any practical applications.

It does. Inheritance of mixins is generally used, for example it's heavily used in Django views https://github.com/django/django/blob/master/django/views/generic/edit.py

I'd prefer a way, where I can declare class as a mixin and specify the class it should be used with.

[zero323]

It does. Inheritance of mixins is generally used, for example it's heavily used in Django views https://github.com/django/django/blob/master/django/views/generic/edit.py

I'd prefer a way, where I can declare class as a mixin and specify the class it should be used with.

I am pretty sure we are not talking about the same thing here. I never suggested that either compost ion or hierarchies of mix-ins are not used in practice.

What I am saying is that supporting such cases using transitive semantics is not practical in context of lightweight self annotations (and probably any other case, without explicit hints for all extending classes). In particular in many cases it would mean that such annotation is as good as ingore[attr-defined], which is not good at all.

[ilevkivskyi]

It would effectively mean that the type check on self should be applied only when Baz is instantiated.

No, not even there. We should only give error when foo attribute is accessed on a Baz instance, for example:

Baz().foo()  # Invalid self type "Baz" for "foo", expected "Foo"

This is just Python works, it actually passes Baz() as the self and this doesn't match the declared type.

[ilevkivskyi]

Another important question here is what to do if one wants to use an attribute of the mixin class in the mixin method body. One possible solution is to implicitly assume the type of self is an intersection of the current class and the explicit annotation. TBH I don't like such implicit behavior.
On the other hand the explicit annotation (when needed) would require supporting user-defined intersection types, which we don't support yet. For example, in the original example we would define T = TypeVar("T", bound=Intersection[Foo, Mixin]), or more realistically T = TypeVar("T", bound=Intersection[FooLike, Mixin]), where FooLike is the minimal protocol we want for the "host" class.

This makes me think maybe we should introduce class-only intersections? I don't think one would ever need to intersect a callable and a tuple. Essentially, this class-only intersection would obey the same rules as Type[...]: only proper classes and unions of such are allowed as arguments. This way we could not even introduce a new kind of types internally and use anonymous TypeInfos instead.

I am not sure however about type variables, Type[T] is allowed if the upper bound is allowed. Btw, this is probably the only reason why a separate TypeType kind is needed internally, otherwise we could represent everything using CallableTypes.

@JukkaL what do you think?

[kaste]

Wow, what a great PR. 👏

[NixBiks]

@zero323 what solution did you end up with? One solution could be writing Protocol for all the required attributes and methods your mixin depends on, but it would be nice to be able to use Foo class directly as a requirement.

[zero323]

@mr-bjerre I am afraid that, after some half-hearted attempts of implementing a plug-in, I decided I could live without it. Still, I think it would be a nice thing to have in general.

[adam-urbanczyk]

This issue is closed but the last example still raises (with 910)

m.py:9: error: The erased type of self "m.Foo" is not a supertype of its class "m.MixIn"

Is there a solution to this or is this a wontfix?

For completeness:

from typing import TypeVar

class Foo: ...

T = TypeVar("T", bound=Foo)

class MixIn:
    def foo(self: T) -> T: ...

[ilevkivskyi]

You can make it a protocol:

from typing import TypeVar, Protocol

class Foo(Protocol):
    host_attr: int

T = TypeVar("T", bound=Foo)

class MixIn:
    def foo(self: T) -> T:
        self.host_attr += 1  # OK
        return self
        
class Host(MixIn):
    host_attr: int
    
class BadHost(MixIn):
    host_attr: str
    
Host().foo()  # OK
BadHost().foo()  # Error: Invalid self argument "BadHost" to attribute function "foo" with type "Callable[[T], T]"

[adam-urbanczyk]

Thanks!