Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pytest: Unable to run tests for non-installed modules #15062

Closed
MKuranowski opened this issue Dec 27, 2020 · 19 comments
Closed

Pytest: Unable to run tests for non-installed modules #15062

MKuranowski opened this issue Dec 27, 2020 · 19 comments
Labels
area-testing bug Issue identified by VS Code Team member as probable bug feature-request Request for new features or functionality

Comments

@MKuranowski
Copy link

Environment data

  • VS Code version: 1.15.2
  • Extension version (available under the Extensions sidebar): 2020.12.42445261
  • OS and version: Manjaro (up to date @ 2020-12-27); Linux x64 5.9.11
  • Python version (& distribution if applicable, e.g. Anaconda): 3.8.6
  • Type of virtual environment used (N/A | venv | virtualenv | conda | ...): n/a
  • Relevant/affected Python packages and their versions: Pytest 6.2.1
  • Relevant/affected Python-related VS Code extensions and their versions: (probably not relevant) Pylance 2020.12.2
  • Value of the python.languageServer setting: Pylance

Setup

Consider the following project layout:

├ .vscode/
│  └ settings.json
├ somepkg/
│  └ __init__.py
└ tests/
   └ test_foo.py

The contents of those files don't really matter, expect that tests/test_foo.py imports somepkg.

Example .vscode/settings.json:

{
    "python.testing.pytestArgs": ["."],
    "python.testing.unittestEnabled": false,
    "python.testing.nosetestsEnabled": false,
    "python.testing.pytestEnabled": true
}

Example somepkg/__init__.py:

def foo():
    return "foo"

Example tests/test_foo.py:

import somepkg
def test_foo():
    assert somepkg.foo() == "foo"

With such project layout, it is impossible to setup VS Code to run those tests without also installing somepkg (for example in a venv).

Here are solutions that I have tried:

Case 0: Do nothing

Obviously, this doesn't work, because invoking bare pytest doesn't add current directory to sys.path

VS Code fails to discover tests:

Test Discovery failed: 
Error: ============================= test session starts ==============================
platform linux -- Python 3.8.6, pytest-6.2.1, py-1.9.0, pluggy-0.13.1
rootdir: /home/mikolaj/Documents/VSCode-Pytest-Bug
collected 0 items / 1 error

==================================== ERRORS ====================================
______________________ ERROR collecting tests/test_foo.py ______________________
ImportError while importing test module '/home/mikolaj/Documents/VSCode-Pytest-Bug/tests/test_foo.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib/python3.8/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/test_foo.py:1: in <module>
    import somepkg
E   ModuleNotFoundError: No module named 'somepkg'
=========================== short test summary info ============================
ERROR tests/test_foo.py
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
===================== no tests collected, 1 error in 0.06s =====================

Traceback (most recent call last):
  File "/home/mikolaj/.vscode/extensions/ms-python.python-2020.12.424452561/pythonFiles/testing_tools/run_adapter.py", line 22, in <module>
    main(tool, cmd, subargs, toolargs)
  File "/home/mikolaj/.vscode/extensions/ms-python.python-2020.12.424452561/pythonFiles/testing_tools/adapter/__main__.py", line 100, in main
    parents, result = run(toolargs, **subargs)
  File "/home/mikolaj/.vscode/extensions/ms-python.python-2020.12.424452561/pythonFiles/testing_tools/adapter/pytest/_discovery.py", line 44, in discover
    raise Exception("pytest discovery failed (exit code {})".format(ec))
Exception: pytest discovery failed (exit code 2)

And similarly command pytest fails:

$ pytest
================================================ test session starts ================================================
platform linux -- Python 3.8.6, pytest-6.2.1, py-1.9.0, pluggy-0.13.1
rootdir: /home/mikolaj/Documents/VSCode-Pytest-Bug
collected 0 items / 1 error                                                                                         

====================================================== ERRORS =======================================================
________________________________________ ERROR collecting tests/test_foo.py _________________________________________
ImportError while importing test module '/home/mikolaj/Documents/VSCode-Pytest-Bug/tests/test_foo.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib/python3.8/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/test_foo.py:1: in <module>
    import somepkg
E   ModuleNotFoundError: No module named 'somepkg'
============================================== short test summary info ==============================================
ERROR tests/test_foo.py
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
================================================= 1 error in 0.06s ==================================================

Case 1: python -m pytest

This is the solution from pytest docs. I added "python.testing.pytestPath": "python -m pytest" to .vscode/settings.json and tried to launch tests.

VS Code, however, still fails on discovery.

> /usr/bin/python ~/.vscode/extensions/ms-python.python-2020.12.424452561/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir ~/Documents/VSCode-Pytest-Bug -s --cache-clear .
cwd: ~/Documents/VSCode-Pytest-Bug
Error 2020-12-27 19:55:08: Failed to parse discovered Test [r [Error]: ============================= test session starts ==============================
platform linux -- Python 3.8.6, pytest-6.2.1, py-1.9.0, pluggy-0.13.1
rootdir: /home/mikolaj/Documents/VSCode-Pytest-Bug
collected 0 items / 1 error

==================================== ERRORS ====================================
______________________ ERROR collecting tests/test_foo.py ______________________
ImportError while importing test module '/home/mikolaj/Documents/VSCode-Pytest-Bug/tests/test_foo.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib/python3.8/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/test_foo.py:1: in <module>
    import somepkg
E   ModuleNotFoundError: No module named 'somepkg'
=========================== short test summary info ============================
ERROR tests/test_foo.py
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
===================== no tests collected, 1 error in 0.06s =====================

Traceback (most recent call last):
  File "/home/mikolaj/.vscode/extensions/ms-python.python-2020.12.424452561/pythonFiles/testing_tools/run_adapter.py", line 22, in <module>
    main(tool, cmd, subargs, toolargs)
  File "/home/mikolaj/.vscode/extensions/ms-python.python-2020.12.424452561/pythonFiles/testing_tools/adapter/__main__.py", line 100, in main
    parents, result = run(toolargs, **subargs)
  File "/home/mikolaj/.vscode/extensions/ms-python.python-2020.12.424452561/pythonFiles/testing_tools/adapter/pytest/_discovery.py", line 44, in discover
    raise Exception("pytest discovery failed (exit code {})".format(ec))
Exception: pytest discovery failed (exit code 2)

	at ChildProcess.<anonymous> (/home/mikolaj/.vscode/extensions/ms-python.python-2020.12.424452561/out/client/extension.js:9:567859)
	at Object.onceWrapper (events.js:313:26)
	at ChildProcess.emit (events.js:223:5)
	at maybeClose (internal/child_process.js:1021:16)
	at Socket.<anonymous> (internal/child_process.js:430:11)
	at Socket.emit (events.js:223:5)
	at Pipe.<anonymous> (net.js:664:12)]

And running pytest manually works fine.

$ python -m pytest
================================================ test session starts ================================================
platform linux -- Python 3.8.6, pytest-6.2.1, py-1.9.0, pluggy-0.13.1
rootdir: /home/mikolaj/Documents/VSCode-Pytest-Bug
collected 1 item                                                                                                    

tests/test_foo.py .                                                                                           [100%]

================================================= 1 passed in 0.01s =================================================

Case 2: Absolute PYTHONPATH

Here, two new keys were added to the original settings.json: "python.envFile": "${workspaceFolder}/.vscode/.env" and "terminal.integrated.env.linux": { "PYTHONPATH": "${workspaceFolder}" }; and the .vscode/.env file was created with the following content. The path is hard-coded to circumvent issue #13749.

# Somehow using ${workspaceFolder} or ${env:PWD} doesn't work here
PYTHONPATH = /home/mikolaj/Documents/VSCode-Pytest-Bug/

And now test discovery works! But then, running the tests errors out.

> /usr/bin/python ~/.vscode/extensions/ms-python.python-2020.12.424452561/pythonFiles/pyvsc-run-isolated.py pytest --override-ini junit_family=xunit1 --rootdir ~/Documents/VSCode-Pytest-Bug --junit-xml=/tmp/tmp-12233yWmGcDH2XD1w.xml .
cwd: ~/Documents/VSCode-Pytest-Bug

============================= test session starts ==============================
platform linux -- Python 3.8.6, pytest-6.2.1, py-1.9.0, pluggy-0.13.1
rootdir: /home/mikolaj/Documents/VSCode-Pytest-Bug
collected 0 items / 1 error

==================================== ERRORS ====================================
______________________ ERROR collecting tests/test_foo.py ______________________
ImportError while importing test module '/home/mikolaj/Documents/VSCode-Pytest-Bug/tests/test_foo.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib/python3.8/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/test_foo.py:1: in <module>
    import somepkg
E   ModuleNotFoundError: No module named 'somepkg'
-------------- generated xml file: /tmp/tmp-10236qdrGjj8MkHVf.xml --------------
=========================== short test summary info ============================
ERROR tests/test_foo.py
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
=============================== 1 error in 0.07s ===============================

And of course, pytest works like a charm:

$ PYTHONPATH=/home/mikolaj/Documents/VSCode-Pytest-Bug/ pytest
================================================ test session starts ================================================
platform linux -- Python 3.8.6, pytest-6.2.1, py-1.9.0, pluggy-0.13.1
rootdir: /home/mikolaj/Documents/VSCode-Pytest-Bug
collected 1 item                                                                                                    

tests/test_foo.py .                                                                                           [100%]

================================================= 1 passed in 0.01s =================================================

Case 3: relative PYTHONPATH

This time settings.json looks like this:

{
    "python.testing.pytestArgs": [
        "."
    ],
    "python.testing.unittestEnabled": false,
    "python.testing.nosetestsEnabled": false,
    "python.testing.pytestEnabled": true,
    "python.testing.pytestPath": "python -m pytest",
    "python.envFile": "${workspaceFolder}/.vscode/.env",
    "terminal.integrated.env.linux": {
        "PYTHONPATH": "."
    },
    "python.testing.cwd": "${workspaceFolder}"
}

And .vscode/.env contains only PYTHONPATH=.

VS Code now correctly discovers the tests, but crashes when trying to run them:

> /usr/bin/python ~/.vscode/extensions/ms-python.python-2020.12.424452561/pythonFiles/pyvsc-run-isolated.py pytest --override-ini junit_family=xunit1 --rootdir ~/Documents/VSCode-Pytest-Bug --junit-xml=/tmp/tmp-10236MyRbp3P5xcEN.xml .
cwd: ~/Documents/VSCode-Pytest-Bug

============================= test session starts ==============================
platform linux -- Python 3.8.6, pytest-6.2.1, py-1.9.0, pluggy-0.13.1
rootdir: /home/mikolaj/Documents/VSCode-Pytest-Bug
collected 0 items / 1 error

==================================== ERRORS ====================================
______________________ ERROR collecting tests/test_foo.py ______________________
ImportError while importing test module '/home/mikolaj/Documents/VSCode-Pytest-Bug/tests/test_foo.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib/python3.8/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/test_foo.py:1: in <module>
    import somepkg
E   ModuleNotFoundError: No module named 'somepkg'
-------------- generated xml file: /tmp/tmp-10236whE7Tokmzock.xml --------------
=========================== short test summary info ============================
ERROR tests/test_foo.py
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
=============================== 1 error in 0.07s ===============================

And, yet again, pytest works fine:

$ PYTHONPATH=. pytest                                         
================================================ test session starts ================================================
platform linux -- Python 3.8.6, pytest-6.2.1, py-1.9.0, pluggy-0.13.1
rootdir: /home/mikolaj/Documents/VSCode-Pytest-Bug
collected 1 item                                                                                                    

tests/test_foo.py .                                                                                           [100%]

================================================= 1 passed in 0.01s =================================================

Expected behavior

I would expect that modifications from cases 1, 2 and 3 allow testing non-installed modules.

Related issues

#6891 - that's not the case. starting from case 3, I have created a separate test in tests/test_env.py:

import os

def test_pythonpath():
    assert os.getenv("PYTHONPATH") == "."

And it passes correctly.

#12420 - there "Run tests" work, here it doesn't

@MKuranowski MKuranowski added triage-needed Needs assignment to the proper sub-team bug Issue identified by VS Code Team member as probable bug labels Dec 27, 2020
@ghost ghost removed the triage-needed Needs assignment to the proper sub-team label Jan 4, 2021
@kimadeline
Copy link

kimadeline commented Jan 4, 2021

Hi @MKuranowski, thank you for reaching out and your thorough research!

I can repro your issue, this seems to be a regression (it used to work), marking it as such.

Thank you again for the report!

@kimadeline kimadeline added needs spike Label for issues that need investigation before they can be worked on. regression Bug didn't exist in a previous release and removed triage labels Jan 4, 2021
@kimadeline kimadeline removed their assignment Jan 4, 2021
@nss350
Copy link

nss350 commented Jan 17, 2021

Hi,
I have encountered a similar issue on Windows. I'm developing a package with a directory structure similar to the one described in the original issue. Additionally, I am using a virtual environment and test discovery appears to work without much issue after adding the following:

"python.pythonPath": /path/to/my/venv/python
"python.testing.pytestEnabled": true

Further, adding this setting allows py.test tests to run from command line. Note, pytest is installed in my virtual environment.

"terminal.integrated.env.windows": {
        "PYTHONPATH": "${PYTHONPATH};${workspaceFolder}"
}

However, I am unable to get the tests to run successfully using the testing interface in vs code. They all fail with import errors. I have gone round and round on this, but nothing seems to work. I initially tried adding a .venv file pointing PYTHONPATH to the current working folder which did not work. Next, I set the following settings:

python.analysis.extraPaths
python.autoComplete.extraPaths (I now believe this is deprecated)

Neither of which worked.

Finally, I made a second copy of my project and pointed my .env file to the copied project folder and finally it worked. Therefore, it does seem that the .env is picked up, but there is no way to add the current working directory to the PYTHONPATH or that this is somehow overwritten.

If it helps, I can provide a more complete example, but it would be quite similar to the one posted here. Note that the Python Text Explorer extension appears to find everything appropriately.

Is there any update on this issue?

@kimadeline
Copy link

Hi @nss350,

No progress on this issue yet, feel free to 👍 it to help bumping up its priority.

Thanks!

@jocelyne8
Copy link

I believe I have a related, but not identical issue. I have been downgrading to extension version v2020.9.111407 (or so) every time it auto-updates to get around it. In my case, the package is installed with pip install -e , so there is an egg-link in the package directory. VS code discovers the tests, the tests complete in debug mode, but give an import error for the package when I run them.

@luabud luabud added the important Issue identified as high-priority label Jan 20, 2021
@luabud
Copy link
Member

luabud commented Jan 27, 2021

Hey @MKuranowski, can you try adding a __init__.py file to your tests folder and see if it works?

@MKuranowski
Copy link
Author

After some time I also found that workaround, so yes, adding tests/__init__.py fixes the issue. Another workaround is to add an empty conftest.py to project root.

@luabud
Copy link
Member

luabud commented Jan 28, 2021

Thanks for checking!

@luabud luabud added needs proposal Need to make some design decisions feature-request Request for new features or functionality and removed important Issue identified as high-priority regression Bug didn't exist in a previous release needs spike Label for issues that need investigation before they can be worked on. labels Jan 28, 2021
@SuaveFool
Copy link

SuaveFool commented Jan 29, 2021

Thanks @luabud I was having the same issue (exactly as described, but in Windows), adding the __init__.py to my tests folder fixes it for me.

@jaron-l
Copy link

jaron-l commented Feb 4, 2021

Forgive me if I don't understand, but is this because pytest run is not obeying the environment variables set in .vscode/.env when running the test? It seems case 3 seems to point to that. This seems related to how PYTHONPATH is set.

@brettcannon
Copy link
Member

@jaron-l I don't think so; "terminal.integrated.env.linux" isn't used by us as we are not running pytest through the terminal.

@jaron-l
Copy link

jaron-l commented Feb 8, 2021

@brettcannon What about "python.envFile"? Not being able to find packages is a python path issue so the different scenarios have different python paths.

@brettcannon
Copy link
Member

@jaron-l if you put the tests/ directory on to sys.path it might make it work (although that also depends on the tests not importing things in such a way that it won't break from that happening).

@phloose
Copy link

phloose commented Feb 11, 2021

I am having the same issue with an editable install (pip install -e). To make the extension run the tests i have to add an __init__.py as @luabud has suggested and others have also successfully tried. Tests discovery works fine without.

I am just wondering why i can easily run pytest from the command line WITHOUT adding an __init__.py in the tests folder. Personally i don't want to clutter the test folder with unnecessary files only to make the extension being able to run the tests.

How does the extension invoke pytest?

@jocelyne8
Copy link

It sounds like it could be the same as this issue: #14579
which is fixed in the dev build.
My solution has been to install v2020.9.114305 every time it upgrades the package.

@phloose
Copy link

phloose commented Feb 16, 2021

As suggested in #14570 (comment) using the user setting "python.useIsolation": false fixes the issue for me.

@Asaurus1
Copy link

Asaurus1 commented Apr 4, 2021

Also having this issue in Version 1.55.0 on Windows x64 and python 3.7.

  • Running python in a virtual env

  • Have installed a model "dalek" in my venv using pip install -e dalek (shows up on pip list) | VSCode testing framework reports "no module named 'dalek'

  • Have a folder 'borg' in my project directory with an __init__.py file in it so it can be imported with import borg. | VSCode testing framework reports "no module named 'borg'"
    image

  • Adding an __init__.py file to my /tests directory as suggested by @luabud DOES seem to work for me.

@EliiseS
Copy link
Member

EliiseS commented Jun 1, 2021

I'm using a devcontainer and I'm having the same issue with my modules. Our tests on the command line also use python -m pytest to run the tests. Adding conftest.py to root of the workspace or __init__.py to tests\ has not worked either for me.

I got the test discovery to work as well by adding "containerEnv": {"PYTHONPATH": "${containerWorkspaceFolder}"}, to the .devcontainer/.devcontainer.json. The Run Test highlight still fails, however, on a very surprising note, debug test does work for running the tests.

@heitorlessa
Copy link

I wrestled with this today and ended up having a .env file with PYTHONPATH=.. This ensured VSCode updated PYTHONPATH to include the current ${workspaceFolder} correctly - just reloaded and worked like a charm.

@eleanorjboyd
Copy link
Member

Hello! I have reviewed this issue and given the architectural changes of the testing rewrite, this issue should no longer exist on the rewrite. Therefore I am going to close this issue but please comment or open a new issue if you are still seeing a problem when you try this yourself!

To use the rewrite yourself just add ”python.experiments.optInto": ["pythonTestAdapter"] to your user settings. You can confirm you have the rewrite enabled by setting your log level to trace, via the Developer: Set Log Level command in the command palette. Then check to see if Experiment 'pythonTestAdapter' is active is in your python logs.

Thanks!

@github-actions github-actions bot removed the needs proposal Need to make some design decisions label Oct 30, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Nov 30, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
area-testing bug Issue identified by VS Code Team member as probable bug feature-request Request for new features or functionality
Projects
None yet
Development

No branches or pull requests