Run mypy on cases_generator

[sobolevn]

Feature or enhancement

After #104421 is merged, we now have the precedent and the needed infrastructure to run mypy on things that support type annotations in Tools/

Pitch

Maybe we should run it on Tools/cases_generator? It has all annotations in place, it needs very little work. Right now mypy finds only 24 errors.

Full list:

» mypy .
parser.py:22: error: Return value expected  [return-value]
                return
                ^~~~~~
parser.py:142: error: Missing return statement  [return]
        def definition(self) -> InstDef | Super | Macro | Family | None:
        ^
parser.py:168: error: Module has no attribute "OVERRIDE"  [attr-defined]
            override = bool(self.expect(lx.OVERRIDE))
                                        ^~~~~~~~~~~
parser.py:169: error: Module has no attribute "REGISTER"  [attr-defined]
            register = bool(self.expect(lx.REGISTER))
                                        ^~~~~~~~~~~
parser.py:177: error: Argument 3 to "InstHeader" has incompatible type "str";
expected "Literal['inst', 'op', 'legacy']"  [arg-type]
    ...                 return InstHeader(override, register, kind, name, inp, outp...
                                                              ^~~~
parser.py:200: error: Incompatible return value type (got
"List[Union[StackEffect, CacheEffect, Node]]", expected
"Optional[List[Union[StackEffect, CacheEffect]]]")  [return-value]
                        return [inp] + rest
                               ^~~~~~~~~~~~
parser.py:202: error: List item 0 has incompatible type "Node"; expected
"Union[StackEffect, CacheEffect]"  [list-item]
                return [inp]
                        ^~~
parser.py:228: error: Missing return statement  [return]
        def cache_effect(self) -> CacheEffect | None:
        ^
parser.py:241: error: Missing return statement  [return]
        def stack_effect(self) -> StackEffect | None:
        ^
parser.py:249: error: Module has no attribute "IF"  [attr-defined]
                if self.expect(lx.IF):
                               ^~~~~
parser.py:284: error: Missing return statement  [return]
        def super_def(self) -> Super | None:
        ^
parser.py:295: error: Missing return statement  [return]
        def ops(self) -> list[OpName] | None:
        ^
parser.py:304: error: Missing return statement  [return]
        def op(self) -> OpName | None:
        ^
parser.py:309: error: Missing return statement  [return]
        def macro_def(self) -> Macro | None:
        ^
parser.py:320: error: Missing return statement  [return]
        def uops(self) -> list[UOp] | None:
        ^
parser.py:328: error: Incompatible return value type (got "List[Node]", expected
"Optional[List[Union[OpName, CacheEffect]]]")  [return-value]
                return uops
                       ^~~~
parser.py:331: error: Missing return statement  [return]
        def uop(self) -> UOp | None:
        ^
parser.py:384: error: Missing return statement  [return]
        def block(self) -> Block | None:
        ^
generate_cases.py:395: error: Item "None" of "Optional[Context]" has no attribute
"owner"  [union-attr]
            filename = context.owner.filename
                       ^~~~~~~~~~~~~
generate_cases.py:598: error: Incompatible types in assignment (expression has type
"Optional[Node]", variable has type "Union[InstDef, Super, Macro, Family, None]") 
[assignment]
            while thing := psr.definition():
                           ^~~~~~~~~~~~~~~~
generate_cases.py:664: error: Item "None" of "Optional[Family]" has no attribute
"name"  [union-attr]
                                f"Instruction {member} is a member of multiple f...
                                ^
generate_cases.py:675: error: Item "None" of "Optional[Family]" has no attribute
"name"  [union-attr]
                                        f"Component {part.instr.name} of macro {...
                                        ^
test_generator.py:46: error: Missing positional argument "metadata_filename" in call
to "Analyzer"  [call-arg]
        a = generate_cases.Analyzer(temp_input.name, temp_output.name)
            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
test_generator.py:46: error: Argument 1 to "Analyzer" has incompatible type "str";
expected "List[str]"  [arg-type]
        a = generate_cases.Analyzer(temp_input.name, temp_output.name)
                                    ^~~~~~~~~~~~~~~
Found 24 errors in 3 files (checked 5 source files)

If @gvanrossum finds it useful, I can surely work on this.
CC @AlexWaygood

Linked PRs

• gh-104504: Run mypy on cases_generator in CI (and blacken the code) #108090
• gh-104504: cases generator: Add --warn-unreachable to the mypy config #108112
• gh-104504: Cases generator: enable mypy's possibly-undefined error code #108454

[sunmy2019]

I think it's useful. Tools/ are for internal usage. The risk is relatively small.

[gvanrossum]

I don't object, and it may occasionally help. I've come this far mostly by using the VS Code type checker (pylance/pyright) in strict mode, and that has definitely helped me. There are also some annoying bits, in particular lexer.py was written in an overly dynamic style. Maybe it should just be fixed. I'm happy to take PRs.

[sobolevn]

Ok then! I will get back to it after the confence I am making is over :)

[gvanrossum]

@sobolevn Are you still planning to work on this? Before you go too deep, I expect that the main source of issues is the dynamic nature of lexer.py. We may just have to refactor that to be less introspective. I think it would improve its maintainability.

[sobolevn]

Yes, it is still on my TODO list. I can increase the priority :)

[corona10]

@sobolevn

Now the issue looks to be solved.
If you want to reopen the issue, please let me know.