Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Crash with recursive types #14000

Closed
dvarrazzo opened this issue Nov 3, 2022 · 3 comments · Fixed by #14007
Closed

Crash with recursive types #14000

dvarrazzo opened this issue Nov 3, 2022 · 3 comments · Fixed by #14007
Labels
crash topic-recursive-types

Comments

@dvarrazzo
Copy link

I just notice #731 closed and I thought giving it a go.

git clone git@github.com:psycopg/psycopg.git
cd psycopg
git reset --hard 82b84e33
python3 -m venv .venv
source .venv/bin/activate
pip install ./psycopg[test,dev]

# Install mypy at release-0.990
pip install "git+https://github.com/python/mypy.git@42a4c6978"

# Enable a recursive type
patch -p1 < HERE
diff --git a/psycopg/psycopg/abc.py b/psycopg/psycopg/abc.py
index 1edde3db..beab963d 100644
--- a/psycopg/psycopg/abc.py
+++ b/psycopg/psycopg/abc.py
@@ -30,9 +30,7 @@ Params: TypeAlias = Union[Sequence[Any], Mapping[str, Any]]
 ConnectionType = TypeVar("ConnectionType", bound="BaseConnection[Any]")
 PipelineCommand: TypeAlias = Callable[[], None]
 
-# TODO: make it recursive when mypy will support it
-# DumperKey: TypeAlias = Union[type, Tuple[Union[type, "DumperKey"]]]
-DumperKey: TypeAlias = Union[type, Tuple[type, ...]]
+DumperKey: TypeAlias = Union[type, Tuple[Union[type, "DumperKey"]]]
 
 # Waiting protocol types
 
HERE

mypy --show-traceback 2>&1 | tee mypy.log

Traceback

psycopg/psycopg/pq/misc.py:99: error: Redundant cast to "PGconn"  [redundant-cast]
psycopg/psycopg/adapt.py:94: error: Return type "Union[type, Tuple[type, ...]]" of "get_key" incompatible with return type "DumperKey" in supertype 
"Dumper"  [override]
Traceback (most recent call last):
  File "/home/piro/dev/fs/mypy/tmp/psycopg/.venv/bin/mypy", line 8, in <module>
    sys.exit(console_entry())
  File "/home/piro/dev/fs/mypy/tmp/psycopg/.venv/lib/python3.8/site-packages/mypy/__main__.py", line 15, in console_entry
    main()
  File "/home/piro/dev/fs/mypy/tmp/psycopg/.venv/lib/python3.8/site-packages/mypy/main.py", line 95, in main
    res, messages, blockers = run_build(sources, options, fscache, t0, stdout, stderr)
  File "/home/piro/dev/fs/mypy/tmp/psycopg/.venv/lib/python3.8/site-packages/mypy/main.py", line 174, in run_build
    res = build.build(sources, options, None, flush_errors, fscache, stdout, stderr)
  File "/home/piro/dev/fs/mypy/tmp/psycopg/.venv/lib/python3.8/site-packages/mypy/build.py", line 193, in build
    result = _build(
  File "/home/piro/dev/fs/mypy/tmp/psycopg/.venv/lib/python3.8/site-packages/mypy/build.py", line 276, in _build
    graph = dispatch(sources, manager, stdout)
  File "/home/piro/dev/fs/mypy/tmp/psycopg/.venv/lib/python3.8/site-packages/mypy/build.py", line 2903, in dispatch
    process_graph(graph, manager)
  File "/home/piro/dev/fs/mypy/tmp/psycopg/.venv/lib/python3.8/site-packages/mypy/build.py", line 3287, in process_graph
    process_stale_scc(graph, scc, manager)
  File "/home/piro/dev/fs/mypy/tmp/psycopg/.venv/lib/python3.8/site-packages/mypy/build.py", line 3388, in process_stale_scc
    graph[id].type_check_first_pass()
  File "/home/piro/dev/fs/mypy/tmp/psycopg/.venv/lib/python3.8/site-packages/mypy/build.py", line 2309, in type_check_first_pass
    self.type_checker().check_first_pass()
  File "/home/piro/dev/fs/mypy/tmp/psycopg/.venv/lib/python3.8/site-packages/mypy/checker.py", line 465, in check_first_pass
    self.accept(d)
  File "/home/piro/dev/fs/mypy/tmp/psycopg/.venv/lib/python3.8/site-packages/mypy/checker.py", line 573, in accept
    stmt.accept(self)
  File "/home/piro/dev/fs/mypy/tmp/psycopg/.venv/lib/python3.8/site-packages/mypy/nodes.py", line 1143, in accept
    return visitor.visit_class_def(self)
  File "/home/piro/dev/fs/mypy/tmp/psycopg/.venv/lib/python3.8/site-packages/mypy/checker.py", line 2132, in visit_class_def
    self.accept(defn.defs)
  File "/home/piro/dev/fs/mypy/tmp/psycopg/.venv/lib/python3.8/site-packages/mypy/checker.py", line 573, in accept
    stmt.accept(self)
  File "/home/piro/dev/fs/mypy/tmp/psycopg/.venv/lib/python3.8/site-packages/mypy/nodes.py", line 1218, in accept
    return visitor.visit_block(self)
  File "/home/piro/dev/fs/mypy/tmp/psycopg/.venv/lib/python3.8/site-packages/mypy/checker.py", line 2569, in visit_block
    self.accept(s)
  File "/home/piro/dev/fs/mypy/tmp/psycopg/.venv/lib/python3.8/site-packages/mypy/checker.py", line 573, in accept
    stmt.accept(self)
  File "/home/piro/dev/fs/mypy/tmp/psycopg/.venv/lib/python3.8/site-packages/mypy/nodes.py", line 822, in accept
    return visitor.visit_func_def(self)
  File "/home/piro/dev/fs/mypy/tmp/psycopg/.venv/lib/python3.8/site-packages/mypy/checker.py", line 938, in visit_func_def
    self._visit_func_def(defn)
  File "/home/piro/dev/fs/mypy/tmp/psycopg/.venv/lib/python3.8/site-packages/mypy/checker.py", line 942, in _visit_func_def
    self.check_func_item(defn, name=defn.name)
  File "/home/piro/dev/fs/mypy/tmp/psycopg/.venv/lib/python3.8/site-packages/mypy/checker.py", line 1011, in check_func_item
    self.check_func_def(defn, typ, name, allow_empty)
  File "/home/piro/dev/fs/mypy/tmp/psycopg/.venv/lib/python3.8/site-packages/mypy/checker.py", line 1208, in check_func_def
    self.accept(item.body)
  File "/home/piro/dev/fs/mypy/tmp/psycopg/.venv/lib/python3.8/site-packages/mypy/checker.py", line 573, in accept

... # 28k lines...

  File "/home/piro/dev/fs/mypy/tmp/psycopg/.venv/lib/python3.8/site-packages/mypy/typestate.py", line 99, in is_assumed_subtype
    if get_proper_type(l) == get_proper_type(left) and get_proper_type(
  File "/home/piro/dev/fs/mypy/tmp/psycopg/.venv/lib/python3.8/site-packages/mypy/types.py", line 2814, in get_proper_type
    typ = typ._expand_once()
  File "/home/piro/dev/fs/mypy/tmp/psycopg/.venv/lib/python3.8/site-packages/mypy/types.py", line 277, in _expand_once
    return replace_alias_tvars(
psycopg/psycopg/_transform.py:73: error: INTERNAL ERROR -- Please try using mypy master on GitHub:
https://mypy.readthedocs.io/en/stable/common_issues.html#using-a-development-mypy-build
Please report a bug at https://github.com/python/mypy/issues
version: 0.990+dev.42a4c69789d855e2a07a5826b4e40331bc407e95
psycopg/psycopg/_transform.py:73: : note: use --pdb to drop into pdb
  File "/home/piro/dev/fs/mypy/tmp/psycopg/.venv/lib/python3.8/site-packages/mypy/types.py", line 3229, in replace_alias_tvars
    new_tp = tp.accept(replacer)
  File "/home/piro/dev/fs/mypy/tmp/psycopg/.venv/lib/python3.8/site-packages/mypy/types.py", line 2592, in accept
    return visitor.visit_union_type(self)
  File "/home/piro/dev/fs/mypy/tmp/psycopg/.venv/lib/python3.8/site-packages/mypy/type_visitor.py", line 273, in visit_union_type
    return UnionType(self.translate_types(t.items), t.line, t.column)
  File "/home/piro/dev/fs/mypy/tmp/psycopg/.venv/lib/python3.8/site-packages/mypy/type_visitor.py", line 276, in translate_types
    return [t.accept(self) for t in types]
  File "/home/piro/dev/fs/mypy/tmp/psycopg/.venv/lib/python3.8/site-packages/mypy/type_visitor.py", line 276, in <listcomp>
    return [t.accept(self) for t in types]
  File "/home/piro/dev/fs/mypy/tmp/psycopg/.venv/lib/python3.8/site-packages/mypy/types.py", line 2172, in accept
    return visitor.visit_tuple_type(self)
  File "/home/piro/dev/fs/mypy/tmp/psycopg/.venv/lib/python3.8/site-packages/mypy/type_visitor.py", line 249, in visit_tuple_type
    self.translate_types(t.items),
  File "/home/piro/dev/fs/mypy/tmp/psycopg/.venv/lib/python3.8/site-packages/mypy/type_visitor.py", line 276, in translate_types
    return [t.accept(self) for t in types]
  File "/home/piro/dev/fs/mypy/tmp/psycopg/.venv/lib/python3.8/site-packages/mypy/type_visitor.py", line 276, in <listcomp>
    return [t.accept(self) for t in types]
  File "/home/piro/dev/fs/mypy/tmp/psycopg/.venv/lib/python3.8/site-packages/mypy/types.py", line 2592, in accept
    return visitor.visit_union_type(self)
  File "/home/piro/dev/fs/mypy/tmp/psycopg/.venv/lib/python3.8/site-packages/mypy/type_visitor.py", line 273, in visit_union_type
    return UnionType(self.translate_types(t.items), t.line, t.column)
  File "/home/piro/dev/fs/mypy/tmp/psycopg/.venv/lib/python3.8/site-packages/mypy/type_visitor.py", line 276, in translate_types
    return [t.accept(self) for t in types]
  File "/home/piro/dev/fs/mypy/tmp/psycopg/.venv/lib/python3.8/site-packages/mypy/type_visitor.py", line 276, in <listcomp>
    return [t.accept(self) for t in types]
  File "/home/piro/dev/fs/mypy/tmp/psycopg/.venv/lib/python3.8/site-packages/mypy/types.py", line 321, in accept
    return visitor.visit_type_alias_type(self)
  File "/home/piro/dev/fs/mypy/tmp/psycopg/.venv/lib/python3.8/site-packages/mypy/types.py", line 3193, in visit_type_alias_type
    return typ.copy_modified(args=[t.accept(self) for t in typ.args])
  File "/home/piro/dev/fs/mypy/tmp/psycopg/.venv/lib/python3.8/site-packages/mypy/types.py", line 354, in copy_modified
    return TypeAliasType(
  File "/home/piro/dev/fs/mypy/tmp/psycopg/.venv/lib/python3.8/site-packages/mypy/types.py", line 262, in __init__
    super().__init__(line, column)
  File "/home/piro/dev/fs/mypy/tmp/psycopg/.venv/lib/python3.8/site-packages/mypy/types.py", line 208, in __init__
    super().__init__(line, column)
RecursionError: maximum recursion depth exceeded while calling a Python object

I tried the same using mypy 0.982 (adding enable_recursive_aliases = true in pyproject.toml) and it segfaults.
@dvarrazzo dvarrazzo added the crash label Nov 3, 2022
@JelleZijlstra
Copy link
Member

cc @ilevkivskyi

@ilevkivskyi ilevkivskyi mentioned this issue Nov 4, 2022
Merged
@ilevkivskyi
Copy link
Member

#14007 should fix this.

Btw @dvarrazzo this type looks suspicious:

  • You don't need unions both inside and outside of the tuple, you can keep only one depending on whether plain type is needed or not.
  • Maybe you are missing ...? Otherwise only valid values are nested 1-element tuples.

Probably you wanted something like

DumperKey: TypeAlias = Union[type, Tuple["DumperKey", ...]]

(which btw is not affected by this bug).

dvarrazzo added a commit to psycopg/psycopg that referenced this issue Nov 4, 2022
@dvarrazzo
fix: use a correct recursive definition for DumperKey
2e924a2 
Already supported in current Mypy: python/mypy#731.

The definition of DumperKey was wrong anyway, in a way causing mypy to
crash: see python/mypy#14000.
@dvarrazzo
Copy link
Author

@ilevkivskyi Thank you for fixing, and you are right: our type definition (a type or an arbitrarily nested tuple of types) was wrong (it was untested, of course). With the type fixed, mypy 0.982 works correctly.

ilevkivskyi added a commit that referenced this issue Nov 4, 2022
@ilevkivskyi
Fix crash on nested unions in recursive types (#14007)
331b170 
Fixes #14000 

This will introduce some minor perf penalty, but only for code that
actually uses recursive types.
svalentin pushed a commit that referenced this issue Nov 7, 2022
@ilevkivskyi @svalentin
Fix crash on nested unions in recursive types (#14007)
91b6fc3 
Fixes #14000 

This will introduce some minor perf penalty, but only for code that
actually uses recursive types.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
crash topic-recursive-types
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix crash on nested unions in recursive types ilevkivskyi/mypy
3 participants
@dvarrazzo @JelleZijlstra @ilevkivskyi