Allow better `tuple` type aliases, fix "Type application" error #12134

sobolevn · 2022-02-06T07:45:18Z

I am also testing how new mypy_primer and mypy_comment actions work: #12125

I expect "No change" comment to appear.

hauntsaninja

Cool! Could you also add a test for type alias my_tuple = tuple; my_tuple[int, float] and for use of cast?

Also IIRC python eval actually runs code with the python interpreter. Would it make more sense for this to be in check-python310.test?

sobolevn · 2022-02-06T08:48:41Z

How is that related? 🤔

 _______________________ testTupleWithDifferentArgsPy310 ________________________
[gw0] linux -- Python 3.10.2 /home/runner/work/mypy/mypy/.tox/py/bin/python
data: /home/runner/work/mypy/mypy/test-data/unit/pythoneval.test:1590:
/home/runner/work/mypy/mypy/mypy/test/testpythoneval.py:44: in run_case
    test_python_evaluation(testcase, os.path.join(self.cache_dir.name, '.mypy_cache'))
/home/runner/work/mypy/mypy/mypy/test/testpythoneval.py:108: in test_python_evaluation
    assert_string_arrays_equal(adapt_output(testcase), output,
E   AssertionError: Invalid output (/home/runner/work/mypy/mypy/test-data/unit/pythoneval.test, line 1590)
----------------------------- Captured stderr call -----------------------------
Expected:
  _testTupleWithDifferentArgsPy310.py:12: error: Invalid type: try using Literal[1] instead? (diff)
  _testTupleWithDifferentArgsPy310.py:13: error: Unexpected "..." (diff)
Actual:
  message.pyi:13: error: Cannot determine type of "Any" (diff)
  _testTupleWithDifferentArgsPy310.py:12: error: Invalid type: try using Literal[1] instead? (diff)
  _testTupleWithDifferentArgsPy310.py:13: error: Unexpected "..." (diff)

AlexWaygood · 2022-02-06T09:40:04Z

message.pyi:13: error: Cannot determine type of "Any" (diff)

Uh, that looks like python/typeshed#7019... Maybe syncing typeshed again might help??

sobolevn · 2022-02-06T09:43:43Z

I think that we should fix it instead of just hiding. I am looking into it 🙂

sobolevn · 2022-02-06T15:03:47Z

It happens here:

mypy/mypy/checkexpr.py

Lines 274 to 275 in 48dc990

    
           if not var.is_ready and self.chk.in_checked_function(): 
        
               self.chk.handle_cannot_determine_type(var.name, context)

sobolevn · 2022-02-06T15:44:37Z

I reduced email/message.pyi to this:

from typing import Any

_HeaderType = Any

class Message:
    def __getitem__(self, name: str) -> _HeaderType: ...

And it still happens.

sobolevn · 2022-02-06T17:21:33Z

It took me literally two hours of debugging. I was able to reproduce this problem only in my newly written test. That's it. Nothing else worked.

Somehow I figured it out (at least I hope so).

AlexWaygood · 2022-02-06T17:26:29Z

It took me literally two hours of debugging. I was able to reproduce this problem only in my newly written test. That's it. Nothing else worked.

Somehow I figured it out (at least I hope so).

Fantastic work - well done! But, bizarre that mypy suddenly started reporting this error recently, after it was fine with this line for so long 🤯

sobolevn · 2022-02-06T17:33:29Z

Thanks, but I still have to fix tuple error, it is also quite cryptic 😒

sobolevn · 2022-02-06T17:56:50Z

mypy_primer seemed to find a similar issue: https://github.com/aio-libs/aiohttp/blob/f78c128f52f4f7e55ebaa8ed0aa764031e98dfe8/aiohttp/web.py#L283

sobolevn · 2022-02-07T07:38:24Z

Ok, here's what is going on. We have a correct test failure:

________________________ testTypeAliasWithBuiltinTuple _________________________
[gw1] linux -- Python 3.8.12 /home/runner/work/mypy/mypy/.tox/py/bin/python
data: /home/runner/work/mypy/mypy/test-data/unit/check-generic-alias.test:243:
/home/runner/work/mypy/mypy/mypy/test/testcheck.py:140: in run_case
    self.run_case_once(testcase)
/home/runner/work/mypy/mypy/mypy/test/testcheck.py:227: in run_case_once
    assert_string_arrays_equal(output, a, msg.format(testcase.file, testcase.line))
E   AssertionError: Unexpected type checker output (/home/runner/work/mypy/mypy/test-data/unit/check-generic-alias.test, line 243)
----------------------------- Captured stderr call -----------------------------
Expected:
  main:6: error: Incompatible types in assignment (expression has type "Tuple...
  main:10: error: Incompatible types in assignment (expression has type "Tupl...
  main:12: note: Revealed type is "builtins.tuple[builtins.int*, ...]" (diff)
Actual:
  main:6: error: Incompatible types in assignment (expression has type "Tuple...
  main:10: error: Incompatible types in assignment (expression has type "Tupl...
  main:12: note: Revealed type is "builtins.tuple[<nothing>, ...]" (diff)

Alignment of first line difference:
  E: ...led type is "builtins.tuple[builtins.int*, ...]"
  A: ...led type is "builtins.tuple[<nothing>, ...]"

Why? Because tuple[int, int]() call gets the original tuple.__new__ signature def [Tco] (iterable: typing.Iterable[Tco] =) -> builtins.tuple[Tco, ...] and tries to replace type vars (Tco) with actual type var values (int, int). apply_generic_arguments does that. It also asserts that the number of original args matches the number of vars to replace.

Right now I skip this step, that's why the test fails.

But, tuple is special, because it is variadic. We need to change this logic with a proper special case.
The problem is: I don't know how to do that.

Should it be def [Tco] (iterable: typing.Iterable[Union[tuple_args]] =) -> builtins.tuple[*tuple_args]? Or something else?

What pyright does in this case? @erictraut I would appreciate your help.

AlexWaygood · 2022-02-13T12:27:34Z

What pyright does in this case? @erictraut I would appreciate your help.

@erictraut any thoughts on this? It looks like pyright handles this kind of thing much better than mypy at the moment — would be great to get your advice, if you've got the time :)

erictraut · 2022-02-13T17:46:20Z

Sorry @AlexWaygood, I missed your earlier question.

I'm not clear on which case you're referring to in your question, but I'll attempt to answer and you can tell me if I misinterpreted your question.

For non-tuple generic classes, pyright maintains a fixed-size list of type arguments once the class is specialized. the length of this list is determined by the fixed number of type parameters in the generic class.

For tuples, pyright maintains a separate (variable-sized) list of type arguments. This is in addition to the fixed-sized list, which corresponds to the type argument for the _T_co type parameter in the tuple class. For example, if it sees tuple[str, int, T], the variable-sized type argument list contains the types [str, int, T] and the fixed-sized list (which has one entry corresponding to _T_co) contains the union of these types [str | int | T].

For the variable-sized tuple type argument list, pyright also records (for each entry) whether the type has a corresponding ... (i.e. whether it represents zero or more of that type). Prior to PEP 646, an entry with a ... could appear only by itself within a tuple because tuple[int, str, ..., None] was not legal. But now with PEP 646, it's legal to spell tuple[int *tuple[str, ...], None] or tuple[int, *Ts, None] where Ts is a TypeVarTuple.

I hope that helps.

JukkaL · 2022-04-14T09:05:56Z

@sobolevn Would you be able to fix the failing tests and the merge conflict so that we can include this in the 0.950 release (#12579)? I can also have a look at finishing this up.

JukkaL · 2022-04-14T09:05:56Z

@sobolevn Would you be able to fix the failing tests and the merge conflict so that we can include this in the 0.950 release (#12579)? I can also have a look at finishing this up.

sobolevn · 2022-04-14T10:38:53Z

@JukkaL sorry, I am not very active right now. I have time to read the inbox, but I don't have much time to actually write code 😢

I hope I will be able to solve all the things in 1-2 months.

github-actions · 2022-05-02T03:24:36Z

Diff from mypy_primer, showing the effect of this PR on open source code:

aiohttp (https://github.com/aio-libs/aiohttp)
+ aiohttp/web.py:283: error: Unused "type: ignore[assignment]" comment

JukkaL · 2022-11-04T16:48:57Z

@sobolevn I'd love to have this fix included in 1.0. If you are busy, I can look into finishing this.

sobolevn · 2022-11-04T18:52:51Z

@JukkaL I would totally appreciate some help on this issue! 👍

JukkaL · 2022-11-25T10:20:45Z

I created #14184 as an updated version of this PR.

@sobolevn

Fix type aliases like these: ``` T = tuple[int, str] ``` Type applications involving fixed-length tuples still don't fully work. The inferred type is a variable-length tuple when constructing a tuple using a type application, e.g. `tuple[int, str]((1, ""))`. This seems a pretty low-priority issue, whereas the type alias use case seems common. Most of the work was by @sobolevn originally in #12134. I just finished it up. Fixes #11098.

Allow better tuple type aliases, fix "Type application" error

7311b74

sobolevn requested a review from hauntsaninja February 6, 2022 07:45

sobolevn mentioned this pull request Feb 6, 2022

(🐞) False positive "error: Type application has too many types (1 expected)" on tuple when annotated as a type or in a TypeAlias with new union syntax (PEP 604) #11098

Closed

hauntsaninja reviewed Feb 6, 2022

View reviewed changes

This comment has been minimized.

Sign in to view

sobolevn mentioned this pull request Feb 6, 2022

Improve readline typings python/typeshed#7141

Merged

Fix Any problem

70ecb89

This comment has been minimized.

Sign in to view

AlexWaygood mentioned this pull request Mar 12, 2022

Cherry pick the typeshed fix for 12339 #12340

Merged

This was referenced Mar 22, 2022

Add _typeshed.Incomplete python/typeshed#7535

Merged

mypy fails for a simple Any type alias` #12421

Closed

JukkaL mentioned this pull request Mar 23, 2022

Fix "Cannot determine type of Any" issue in typeshed #12429

Closed

Merge branch 'master' into issue-11098

83155aa

hauntsaninja reopened this Jul 28, 2022

JukkaL added a commit that referenced this pull request Nov 25, 2022

Add test cases from #12134 by @sobolevn

7bb0cdc

JukkaL mentioned this pull request Nov 25, 2022

Fix type aliases with fixed-length tuples #14184

Merged

JukkaL closed this Nov 25, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow better `tuple` type aliases, fix "Type application" error #12134

Allow better `tuple` type aliases, fix "Type application" error #12134

sobolevn commented Feb 6, 2022 •

edited

Loading

hauntsaninja left a comment

This comment has been minimized.

sobolevn commented Feb 6, 2022

AlexWaygood commented Feb 6, 2022

sobolevn commented Feb 6, 2022

sobolevn commented Feb 6, 2022

sobolevn commented Feb 6, 2022

sobolevn commented Feb 6, 2022 •

edited

Loading

AlexWaygood commented Feb 6, 2022

sobolevn commented Feb 6, 2022

This comment has been minimized.

sobolevn commented Feb 6, 2022

sobolevn commented Feb 7, 2022 •

edited

Loading

AlexWaygood commented Feb 13, 2022

erictraut commented Feb 13, 2022 •

edited

Loading

JukkaL commented Apr 14, 2022

JukkaL commented Apr 14, 2022

sobolevn commented Apr 14, 2022

github-actions bot commented May 2, 2022

JukkaL commented Nov 4, 2022

sobolevn commented Nov 4, 2022

JukkaL commented Nov 25, 2022

Allow better tuple type aliases, fix "Type application" error #12134

Allow better tuple type aliases, fix "Type application" error #12134

Conversation

sobolevn commented Feb 6, 2022 • edited Loading

hauntsaninja left a comment

Choose a reason for hiding this comment

This comment has been minimized.

sobolevn commented Feb 6, 2022

AlexWaygood commented Feb 6, 2022

sobolevn commented Feb 6, 2022

sobolevn commented Feb 6, 2022

sobolevn commented Feb 6, 2022

sobolevn commented Feb 6, 2022 • edited Loading

AlexWaygood commented Feb 6, 2022

sobolevn commented Feb 6, 2022

This comment has been minimized.

sobolevn commented Feb 6, 2022

sobolevn commented Feb 7, 2022 • edited Loading

AlexWaygood commented Feb 13, 2022

erictraut commented Feb 13, 2022 • edited Loading

JukkaL commented Apr 14, 2022

JukkaL commented Apr 14, 2022

sobolevn commented Apr 14, 2022

github-actions bot commented May 2, 2022

JukkaL commented Nov 4, 2022

sobolevn commented Nov 4, 2022

JukkaL commented Nov 25, 2022

Allow better `tuple` type aliases, fix "Type application" error #12134

Allow better `tuple` type aliases, fix "Type application" error #12134

sobolevn commented Feb 6, 2022 •

edited

Loading

sobolevn commented Feb 6, 2022 •

edited

Loading

sobolevn commented Feb 7, 2022 •

edited

Loading

erictraut commented Feb 13, 2022 •

edited

Loading