Support Python 3.6 attribute annotations.

[Tinche]

Python 3.6 allows this:

@attr.s
class A:
    a: int = attr.ib()

The annotation is available as A.__annotations__['a']. So, I suggest that on 3.6, we add an attribute to attr.Attribute called annotation and expose these values there if they are present.

This might be an easy one for someone looking to get started contributing.

[hynek]

Feel free to use the according label generously. :)

[DRMacIver]

I'm having a look into this. Some brief thoughts and questions:

• I think it's important to support annotation on < 3.6 if you're going to support it on 3.6. How do you feel about an a = attr.ib(annotation=int) as an alternate API that does the same thing?
• As well as exposing it on the Attribute object this feels like it should really be made available as standard type annotations on various methods. I can think of three places where the information needs to be propagated: init, property setter, and property getter. Can you think of anywhere else?
• Propagating the type to methods is arguably a breaking change because it could cause code that currently type checks to fail type checking. Do you consider it to be one?
• If yes, hiding it behind an off by default propagate_types flag would be sufficient. This seems like a good thing to have anyway, though my preference would be on by default.
• How do you want to test Python 3.6 specific code? Currently your tests are all in a single undifferentiated directory.

[DRMacIver]

Actually, here's a more fundamental problem with this proposal which I think means it can't/shouldn't work as written: It's flat out lying to the type checker.

In the provided example you've declared that A.a should be an int, but it's actually an Attribute object. Thus correct code using it won't type check, and incorrect code using it will.

[hynek]

Seems like we need a different approach.

[Tinche]

Ok, I'm a little upset this was closed without at least waiting on my input.

Note that type annotations have other uses than mypy. attrs does declarative classes, which is a foreign concept to core Python, so of course there will be some friction, but for example

a = attr.ib()  # type: int

works just fine with the PyCharm type checker. And how is it lying to the type checker? It's setting the type checker straight.

[Tinche]

In the provided example you've declared that A.a should be an int, but it's actually an Attribute object. Thus correct code using it won't type check, and incorrect code using it will.

Actually, I'd have to had used ClassVar to have done that. In reality, A.a won't be an Attribute object but won't exist if we're dealing with dict classes or be a special descriptor if dealing with slot classes. A(1).a will be an int, though, and this is what's claimed by the annotation.

Also note this example from the official Python docs:

class Employee(NamedTuple):
    name: str
    id: int

Lying to the type checker too?

[hynek]

Uh I'm sorry it looked uncontroversial to me. I'll have a deeper look in the next days. Please everyone make your points. :)

[hynek]

(I should stop making rushed decisions on my phone. 😞)

[DRMacIver]

Note that type annotations have other uses than mypy.

Sure, but a number of those other uses are also type checking. e.g. pycharm. I think using them for things other than type checking is great, but using them in ways that are incompatible with type checking is problematic.

Lying to the type checker too?

No, because there are no values assigned that are not of the claimed type (though I admit it's weird).

[ambv]

I think Tinche's syntax suggestion, e.g. using variable annotations / # type: comments is more practical than #162. It already does useful things with existing type checkers today whereas no tool understand annotation= at all.

We'll need some plumbing to make it work. Would be good for 3.6+ if attrs preserved __annotations__ for generated methods if it can read them from variable annotations. Would be great to add type checker support for the @attr.s class decorator. I'm eager to implement this.

But first let's address @DRMacIver's concerns around lying to the type checker. Let's look at what mypy will actually do.

Attribute use on instances

mypy will do the intuitive thing here. Consider the following snippet:

@attr.s()
class Yeah:
    name: str = attr.ib()
    age: int = attr.ib(convert=int, default=0)


def main() -> None:
    y = Yeah('Sue', '16')
    print(y.name.upper())
    print('Next year:', y.age + 1)

First of all look at the snippet again. It looks obvious to human readers, at least the ones familiar with attrs. They understand the attr.ib() plumbing is for instances. They won't have trouble understanding the type annotation behaves the same. I also don't think anybody will be surprised that the class attribute is an Attribute instance.

But let's get back to mypy. It type-checks the attribute usage in main() cleanly. The only thing it's complaining about is the magic __init__ accepting arguments:

example.py:11: error: Too many arguments for "Yeah"

But try flipping the attributes instead in main():

    print(y.age.upper())
    print('Next year:', y.name + 1)

and you'll see it complain about invalid operations, as you'd expect:

example.py:11: error: Too many arguments for "Yeah"
example.py:12: error: "int" has no attribute "upper"
example.py:13: error: Unsupported operand types for + ("str" and "int")

So far, so good.

Attribute use on classes

Here mypy gets it wrong. If we want to access Attribute objects on the class, mypy will raise false positives. print(Yeah.name.validator) will produce:

example.py:15: error: "str" has no attribute "validator"

If we try to mistakenly access the class attribute as an instance attribute (for example by treating Yeah.name as a string, like print(Yeah.name.upper())), then mypy doesn't generate a warning. The program fails at runtime.

Summing up

Even though mypy will get the class-level access wrong, I still prefer using type annotations because:

• it is already immediately useful in the common case; and
• it reads better to human readers.

The fundamental issue is that @attr.s is a class decorator that defines or redefines quite a bunch of methods that aren't there on the original class. The type checker can't see those anyway. So, all in all, we need to teach the type checker about the class decorator so that:

• it understands our __init__ (and other?) generated methods better;
• we silence false positives around Attribute object usage;
• it will raise errors when users try to access class attributes like instance attributes.

The first one is the most important because it generates false positives in the common case.
The second one is less important because it generates false positives in the uncommon case. It also can be worked around with casting.
The third one is the least important because it's simply silent. Users of gradual typing expect that the absence of generated errors is no proof of correctness.

[Tinche]

I basically agree with @ambv but I'm going to share my two Croatian lipa anyway.

In order to get full mypy support we're almost certainly going to have to do extra work, like writing a mypy plugin when that's available. There's probably no chance mypy is going to be competent enough to grok our __init__ construction code by itself.

I see the main bone of contention here is that the core premise of attrs is you write one thing in order to declare another. So this would be the proper way of writing a class:

@attr.s
class A:
    a: ClassVar[_CountingAttr] = attr.ib(annotation=int)

This describes exactly what's happening while you're inside the class, defining the attributes. The only benefit this has is it might help you if you're using another function, like a wrapper for attr.ib(), and you're using it incorrectly - maybe you mistyped a function name, and it doesn't return a _CountingAttr instance but an int, and then mypy will complain. Or if you by mistake use something else other than attr.ib().

@attr.s
class A:
    a: ClassVar[_CountingAttr] = cattr.typed(annotation=int)   # cattr.typed actually returns something else! And mypy might catch this!

The downsides are it's kind of misleading you, the reader. When attr.s finishes processing the class, A.a won't actually be a _CountingAttr. In the case of slot classes, it will be a slot descriptor (a Python internal thing), and in the case of dict classes it won't exist (see http://attrs.readthedocs.io/en/stable/changelog.html#deprecations, the fact it exists now is deprecated and cleanup logic will be added).

The proposed approach,

@attr.s
class A:
    a: int = attr.ib()

is, yes, basically lying to the type checker. But I would argue it's intuitively much clearer to human readers.

Also consider that the point of attrs is reducing boilerplate and making writing classes fun, and compare the two approaches:

@attr.s
class A:
    a: int = attr.ib()
    b: ClassVar[_CountingAttr] = attr.ib(annotation=int)

One is, imo, more beautiful/readable than the other :)

[hynek]

yep, b is the clear winner here. ;P

Trolling aside, it’s my understanding, that we can’t really move on, until python/mypy#2088 is fixed ~somehow~, correct?

[Tinche]

Ok, let's put this on hold until the mypy situation is clearer.

[agronholm]

Couldn't we simply add a variable when its annotation is found? Like simply a: int ? Do we really need the extra boilerplate of = attr.ib()?

[Screwtapello]

It may be instructive to review pydantic, a vaguely similar library that does support Python 3 type-annotations for data validation.

[chadrik]

I added a PR for pyi stubs: #238. I did not see this issue first, but I independently came to the same conclusion as @Tinche that we had to lie about the actual type returned by attr.ib. See my notes there, if you're interested.

Also relevant is this PR for basic type support: #239

[hynek]

This has been fixed by #239 although there are some questions open as to what to do with the information. That discussions should happen on #215.