Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Recursion Error #65

Closed
jonyscathe opened this issue Aug 17, 2023 · 8 comments · Fixed by #71
Closed

Recursion Error #65

jonyscathe opened this issue Aug 17, 2023 · 8 comments · Fixed by #71

Comments

@jonyscathe
Copy link

On a couple of repositories I am getting a recursion error from pydoclint in flake8.

I am running pydoclint with the following settings in my pyproject.toml tool.flake8 section:

skip-checking-short-docstrings = false
style = 'sphinx'

If I remove both of those settings (but not one...) then I do not get a recursion error.
If I remove skip-checking-short-docstrings and change style to google then I do not get a recursion error.

This is only happening on a couple of projects. I have not yet worked out what makes this happen or not happen on other projects with the same settings.
I will try and work out what exactly is causing this tomorrow.

The trace of the error I get is:

pre-commit run -a flake8
flake8...................................................................Failed
- hook id: flake8
- exit code: 1

multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/home/jforrest/.pyenv/versions/3.11.4/lib/python3.11/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
                    ^^^^^^^^^^^^^^^^^^^
  File "/home/jforrest/.cache/pre-commit/repojjx2ocym/py_env-python3.11/lib/python3.11/site-packages/flake8/checker.py", line 82, in _mp_run
    ).run_checks()
      ^^^^^^^^^^^^
  File "/home/jforrest/.cache/pre-commit/repojjx2ocym/py_env-python3.11/lib/python3.11/site-packages/flake8_noqa/noqa_filter.py", line 189, in run_checks
    result = super().run_checks(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jforrest/.cache/pre-commit/repojjx2ocym/py_env-python3.11/lib/python3.11/site-packages/flake8/checker.py", line 525, in run_checks
    self.run_ast_checks()
  File "/home/jforrest/.cache/pre-commit/repojjx2ocym/py_env-python3.11/lib/python3.11/site-packages/flake8/checker.py", line 427, in run_ast_checks
    for line_number, offset, text, _ in runner:
  File "/home/jforrest/.cache/pre-commit/repojjx2ocym/py_env-python3.11/lib/python3.11/site-packages/pydoclint/flake8_entry.py", line 218, in run
    v.visit(self._tree)
  File "/home/jforrest/.pyenv/versions/3.11.4/lib/python3.11/ast.py", line 418, in visit
    return visitor(node)
           ^^^^^^^^^^^^^
  File "/home/jforrest/.pyenv/versions/3.11.4/lib/python3.11/ast.py", line 426, in generic_visit
    self.visit(item)
  File "/home/jforrest/.pyenv/versions/3.11.4/lib/python3.11/ast.py", line 418, in visit
    return visitor(node)
           ^^^^^^^^^^^^^
  File "/home/jforrest/.cache/pre-commit/repojjx2ocym/py_env-python3.11/lib/python3.11/site-packages/pydoclint/visitor.py", line 68, in visit_ClassDef
    self.generic_visit(node)
  File "/home/jforrest/.pyenv/versions/3.11.4/lib/python3.11/ast.py", line 426, in generic_visit
    self.visit(item)
  File "/home/jforrest/.pyenv/versions/3.11.4/lib/python3.11/ast.py", line 418, in visit
    return visitor(node)
           ^^^^^^^^^^^^^
  File "/home/jforrest/.cache/pre-commit/repojjx2ocym/py_env-python3.11/lib/python3.11/site-packages/pydoclint/visitor.py", line 127, in visit_FunctionDef
    argViolations = self.checkArguments(node, parent_, doc)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jforrest/.cache/pre-commit/repojjx2ocym/py_env-python3.11/lib/python3.11/site-packages/pydoclint/visitor.py", line 315, in checkArguments
    astArgList: List[ast.arg] = collectFuncArgs(node)
                                ^^^^^^^^^^^^^^^^^^^^^
  File "/home/jforrest/.cache/pre-commit/repojjx2ocym/py_env-python3.11/lib/python3.11/site-packages/pydoclint/utils/generic.py", line 32, in collectFuncArgs
    kwarg = copy.deepcopy(node.args.kwarg)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jforrest/.pyenv/versions/3.11.4/lib/python3.11/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jforrest/.pyenv/versions/3.11.4/lib/python3.11/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
            ^^^^^^^^^^^^^^^^^^^^^
  File "/home/jforrest/.pyenv/versions/3.11.4/lib/python3.11/copy.py", line 146, in deepcopy
    y = copier(x, memo)
        ^^^^^^^^^^^^^^^
  File "/home/jforrest/.pyenv/versions/3.11.4/lib/python3.11/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
                             ^^^^^^^^^^^^^^^^^^^^^
  File "/home/jforrest/.pyenv/versions/3.11.4/lib/python3.11/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jforrest/.pyenv/versions/3.11.4/lib/python3.11/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
            ^^^^^^^^^^^^^^^^^^^^^


-------------------------------------- This continues for a very long time ---------------------------------------

 File "/home/jforrest/.pyenv/versions/3.11.4/lib/python3.11/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jforrest/.pyenv/versions/3.11.4/lib/python3.11/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
            ^^^^^^^^^^^^^^^^^^^^^
  File "/home/jforrest/.pyenv/versions/3.11.4/lib/python3.11/copy.py", line 146, in deepcopy
    y = copier(x, memo)
        ^^^^^^^^^^^^^^^
  File "/home/jforrest/.pyenv/versions/3.11.4/lib/python3.11/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
                             ^^^^^^^^^^^^^^^^^^^^^
  File "/home/jforrest/.pyenv/versions/3.11.4/lib/python3.11/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jforrest/.pyenv/versions/3.11.4/lib/python3.11/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
            ^^^^^^^^^^^^^^^^^^^^^
  File "/home/jforrest/.pyenv/versions/3.11.4/lib/python3.11/copy.py", line 146, in deepcopy
    y = copier(x, memo)
        ^^^^^^^^^^^^^^^
  File "/home/jforrest/.pyenv/versions/3.11.4/lib/python3.11/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
                             ^^^^^^^^^^^^^^^^^^^^^
  File "/home/jforrest/.pyenv/versions/3.11.4/lib/python3.11/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jforrest/.pyenv/versions/3.11.4/lib/python3.11/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
            ^^^^^^^^^^^^^^^^^^^^^
  File "/home/jforrest/.pyenv/versions/3.11.4/lib/python3.11/copy.py", line 146, in deepcopy
    y = copier(x, memo)
        ^^^^^^^^^^^^^^^
  File "/home/jforrest/.pyenv/versions/3.11.4/lib/python3.11/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
                             ^^^^^^^^^^^^^^^^^^^^^
  File "/home/jforrest/.pyenv/versions/3.11.4/lib/python3.11/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jforrest/.pyenv/versions/3.11.4/lib/python3.11/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
            ^^^^^^^^^^^^^^^^^^^^^
  File "/home/jforrest/.pyenv/versions/3.11.4/lib/python3.11/copy.py", line 137, in deepcopy
    d = id(x)
        ^^^^^
RecursionError: maximum recursion depth exceeded while calling a Python object
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/jforrest/.cache/pre-commit/repojjx2ocym/py_env-python3.11/bin/flake8", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/jforrest/.cache/pre-commit/repojjx2ocym/py_env-python3.11/lib/python3.11/site-packages/flake8/main/cli.py", line 23, in main
    app.run(argv)
  File "/home/jforrest/.cache/pre-commit/repojjx2ocym/py_env-python3.11/lib/python3.11/site-packages/flake8/main/application.py", line 198, in run
    self._run(argv)
  File "/home/jforrest/.cache/pre-commit/repojjx2ocym/py_env-python3.11/lib/python3.11/site-packages/flake8/main/application.py", line 187, in _run
    self.run_checks()
  File "/home/jforrest/.cache/pre-commit/repojjx2ocym/py_env-python3.11/lib/python3.11/site-packages/flake8/main/application.py", line 103, in run_checks
    self.file_checker_manager.run()
  File "/home/jforrest/.cache/pre-commit/repojjx2ocym/py_env-python3.11/lib/python3.11/site-packages/flake8/checker.py", line 235, in run
    self.run_parallel()
  File "/home/jforrest/.cache/pre-commit/repojjx2ocym/py_env-python3.11/lib/python3.11/site-packages/flake8/checker.py", line 204, in run_parallel
    self.results = list(pool.imap_unordered(_mp_run, self.filenames))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jforrest/.pyenv/versions/3.11.4/lib/python3.11/multiprocessing/pool.py", line 873, in next
    raise value
RecursionError: maximum recursion depth exceeded while calling a Python object
@jsh9
Copy link
Owner

jsh9 commented Aug 17, 2023

You can try to use flake8 on selected Python scripts to isolate the file(s) faster.

@jonyscathe
Copy link
Author

I did do that today.
I narrowed it down to a fairly complicated class that had kwargs as an argument which triggers this recursion depth exceeded.
When having a look at what was being copied as part of the deepcopy of node.args.kwarg it ends up copying basically the entire class one line at a time recursively (or at least any lines that can link back to the use of kwargs).
I'm really not sure what the best solution is here (well, other than some code refactoring on my part).
Is there any reason here to do a deepcopy rather than the standard shallow copy?
With a shallow copy the kwarg.arg = '**' + kwarg.arg line still does not modify the original node.args.kwarg's arg value as that is not a compound object. And I don't believe that kwarg is being modified in other way, especially not any contained compound objects.

@jsh9
Copy link
Owner

jsh9 commented Aug 18, 2023

Hi @jonyscathe , without a concrete reproducible case, it may be hard for me to visualize what exactly happened to you.

You can narrow the issue down to a function/method (if it's a method in a class, you can remove the other methods; the reproducible case doesn't even have to be syntactically correct -- it can have uninitialized names).

@jonyscathe
Copy link
Author

I have been trying to reproduce this in a simpler case but have not been able to so far.
It looks like as soon as I remove any complexity from the class at all then I no longer get recursion errors, so that is possibly my best way forward.

@jsh9
Copy link
Owner

jsh9 commented Aug 18, 2023 •
edited
Loading

I suppose you cannot share the code of that whole class with me? (If you'd like to share it in private, you can email it to me. My email is: ...

If you'd rather not share that code, maybe you can try to replace deepcopy with copy and see if that solves your problem.

This section in the documentation can hopefully help you walk through the code base (with breakpoint): https://jsh9.github.io/pydoclint/notes_for_developers.html#2-how-to-quickly-sanity-check-a-code-example

@jonyscathe
Copy link
Author

Replacing deepcopy with copy does definitely solve my problem.

Unfortunately I can't share that class with anyone without violating an NDA so I'm not able to do that.

If replacing deepcopy with copy isn't something you want to do, then we can just close this issue. This class definitely needs some refactoring to reduce complexity anyway so this is a good reason to do that.

@jsh9 jsh9 mentioned this issue Aug 19, 2023
Merged
@jsh9 jsh9 closed this as completed in #71 Aug 19, 2023
@jsh9
Copy link
Owner

jsh9 commented Aug 19, 2023

Hi @jonyscathe , I made a code change to use try/catch. I'll first use deepcopy() and if that triggers RecursionError, it will fall back to copy().

@jsh9
Copy link
Owner

jsh9 commented Aug 19, 2023

This change is included in the new published version, 0.2.0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Deal with potential recursion error jsh9/pydoclint
2 participants
@jonyscathe @jsh9