Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pytest >=8.1.1 displays no diff for AssertionError with --import-mode=importlib #12659

Closed
devanubis opened this issue Jul 24, 2024 · 2 comments · Fixed by #12716
Closed

pytest >=8.1.1 displays no diff for AssertionError with --import-mode=importlib #12659

devanubis opened this issue Jul 24, 2024 · 2 comments · Fixed by #12716

Comments

@devanubis
Copy link

devanubis commented Jul 24, 2024

Description

It seems that from pytest 8.1.1 onwards, using --import-mode=importlib causes AssertionError to not render a diff in the shell output.

8.0.2 worked just fine (as did 7.x).

To reproduce, the test has to be within a package with a __init__.py present (which our tests are, for reasons).

Reproduction

A clean environment:

python -m venv .venv
source .venv/bin/activate
pip install pytest==8.1.1
mkdir tests
touch tests/__init__.py

And then put a simple test file in the tests dir:

def test():
    assert "four lights" == "five lights"

And then run: pytest --import-mode=importlib tests/test.py

Outputs:

=========================================== test session starts ============================================
platform linux -- Python 3.11.2, pytest-8.1.1, pluggy-1.5.0
rootdir: /home/.../pytest-diff-reproduction
collected 1 item                                                                                           

tests/test.py F                                                                                      [100%]

================================================= FAILURES =================================================
___________________________________________________ test ___________________________________________________

    def test():
>       assert "four lights" == "five lights"
E       AssertionError

tests/test.py:2: AssertionError
========================================= short test summary info ==========================================
FAILED tests/test.py::test - AssertionError
============================================ 1 failed in 0.01s =============================================

Without a diff after the AssertionError

If ran without --import-mode=importlib on 8.0.2 the output shows the diff:

=========================================== test session starts ============================================
platform linux -- Python 3.11.2, pytest-8.1.1, pluggy-1.5.0
rootdir: /home/.../pytest-diff-reproduction
collected 1 item                                                                                           

tests/test.py F                                                                                      [100%]

================================================= FAILURES =================================================
___________________________________________________ test ___________________________________________________

    def test():
>       assert "four lights" == "five lights"
E       AssertionError: assert 'four lights' == 'five lights'
E         
E         - five lights
E         + four lights

tests/test.py:2: AssertionError
========================================= short test summary info ==========================================
FAILED tests/test.py::test - AssertionError: assert 'four lights' == 'five lights'
============================================ 1 failed in 0.02s =============================================

Verbosity has no effect, no other packages need to be present (although setuptools etc from the default venv are).

Environment

Debian 12, python 3.11 (but also reproduced on 3.12 too)

@dongfangtianyu
Copy link
Contributor

dongfangtianyu commented Aug 12, 2024

To be more precise, this phenomenon occurs in pytest >8.1.0.

Judging by the behavior, it seems that the assertion rewriting is not working.
From the code, it appears that some changes in pathlib might be causing this.

I am currently debugging further to provide more useful information.


@dongfangtianyu
Copy link
Contributor

dongfangtianyu commented Aug 14, 2024

Reproduction Conditions:

  1. pytest >= 8.1.0
  2. Use the --import-mode=importlib parameter
  3. The directory containing the test cases has an __init__.py file

Execution Process:

  1. Attempt to resolve pkg_path, pkg_root, and module_name (using _pytest.pathlib.resolve_pkg_root_and_module_name)
  2. If resolution is successful, call the function _import_module_using_spec(module_location=pkg_root, insert_modules=False)
  3. If resolution fails, call the function _import_module_using_spec(module_location=path.parent, insert_modules=True)
  4. In _import_module_using_spec, use the hook in sys.meta_path, which includes the assertion rewriting plugin

Error Cause:

  1. pkg_root can be successfully resolved: pkg_root == pkg_path.parent == path.parent.parent
  2. Call _import_module_using_spec(module_location=path.parent.parent, insert_modules=False)
  3. Call AssertionRewritingHook.find_spec, which returns spec=None, causing this method to return early, so assertion rewriting does not occur.

If pkg_root fails to resolve, then _import_module_using_spec(module_location=path.parent, insert_modules=True) will be called, allowing us to obtain the correct spec, which can then be used to smoothly proceed with assertion rewriting.

Update:

AssertionRewritingHook.find_spec uses importlib.machinery.PathFinder.find_spec to retrieve the spec.

Class method that attempts to find a spec for the module specified by fullname on sys.path or, if defined, on path.
https://docs.python.org/3/library/importlib.html#importlib.machinery.PathFinder.find_spec

Based on testing, this method is only suitable for importing modules from directories and is not applicable for importing packages and namespace packages from subdirectories.

dongfangtianyu pushed a commit to dongfangtianyu/pytest that referenced this issue Aug 14, 2024
dongfangtianyu added a commit to dongfangtianyu/pytest that referenced this issue Aug 15, 2024
dongfangtianyu added a commit to dongfangtianyu/pytest that referenced this issue Aug 15, 2024
dongfangtianyu added a commit to dongfangtianyu/pytest that referenced this issue Aug 15, 2024
dongfangtianyu added a commit to dongfangtianyu/pytest that referenced this issue Aug 16, 2024
patchback bot pushed a commit that referenced this issue Aug 30, 2024
nicoddemus pushed a commit that referenced this issue Aug 30, 2024
Fixes #12659

(cherry picked from commit 9a444d1)

Co-authored-by: dongfangtianyu <7629022+dongfangtianyu@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants