Pytest: Unable to run tests for non-installed modules

[MKuranowski]

Environment data

• VS Code version: 1.15.2
• Extension version (available under the Extensions sidebar): 2020.12.42445261
• OS and version: Manjaro (up to date @ 2020-12-27); Linux x64 5.9.11
• Python version (& distribution if applicable, e.g. Anaconda): 3.8.6
• Type of virtual environment used (N/A | venv | virtualenv | conda | ...): n/a
• Relevant/affected Python packages and their versions: Pytest 6.2.1
• Relevant/affected Python-related VS Code extensions and their versions: (probably not relevant) Pylance 2020.12.2
• Value of the python.languageServer setting: Pylance

Setup

Consider the following project layout:

├ .vscode/
│  └ settings.json
├ somepkg/
│  └ __init__.py
└ tests/
   └ test_foo.py

The contents of those files don't really matter, expect that tests/test_foo.py imports somepkg.

Example .vscode/settings.json:

{
    "python.testing.pytestArgs": ["."],
    "python.testing.unittestEnabled": false,
    "python.testing.nosetestsEnabled": false,
    "python.testing.pytestEnabled": true
}

Example somepkg/__init__.py:

def foo():
    return "foo"

Example tests/test_foo.py:

import somepkg
def test_foo():
    assert somepkg.foo() == "foo"

With such project layout, it is impossible to setup VS Code to run those tests without also installing somepkg (for example in a venv).

Here are solutions that I have tried:

Case 0: Do nothing

Obviously, this doesn't work, because invoking bare pytest doesn't add current directory to sys.path

VS Code fails to discover tests:

Test Discovery failed: 
Error: ============================= test session starts ==============================
platform linux -- Python 3.8.6, pytest-6.2.1, py-1.9.0, pluggy-0.13.1
rootdir: /home/mikolaj/Documents/VSCode-Pytest-Bug
collected 0 items / 1 error

==================================== ERRORS ====================================
______________________ ERROR collecting tests/test_foo.py ______________________
ImportError while importing test module '/home/mikolaj/Documents/VSCode-Pytest-Bug/tests/test_foo.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib/python3.8/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/test_foo.py:1: in <module>
    import somepkg
E   ModuleNotFoundError: No module named 'somepkg'
=========================== short test summary info ============================
ERROR tests/test_foo.py
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
===================== no tests collected, 1 error in 0.06s =====================

Traceback (most recent call last):
  File "/home/mikolaj/.vscode/extensions/ms-python.python-2020.12.424452561/pythonFiles/testing_tools/run_adapter.py", line 22, in <module>
    main(tool, cmd, subargs, toolargs)
  File "/home/mikolaj/.vscode/extensions/ms-python.python-2020.12.424452561/pythonFiles/testing_tools/adapter/__main__.py", line 100, in main
    parents, result = run(toolargs, **subargs)
  File "/home/mikolaj/.vscode/extensions/ms-python.python-2020.12.424452561/pythonFiles/testing_tools/adapter/pytest/_discovery.py", line 44, in discover
    raise Exception("pytest discovery failed (exit code {})".format(ec))
Exception: pytest discovery failed (exit code 2)

And similarly command pytest fails:

$ pytest
================================================ test session starts ================================================
platform linux -- Python 3.8.6, pytest-6.2.1, py-1.9.0, pluggy-0.13.1
rootdir: /home/mikolaj/Documents/VSCode-Pytest-Bug
collected 0 items / 1 error                                                                                         

====================================================== ERRORS =======================================================
________________________________________ ERROR collecting tests/test_foo.py _________________________________________
ImportError while importing test module '/home/mikolaj/Documents/VSCode-Pytest-Bug/tests/test_foo.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib/python3.8/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/test_foo.py:1: in <module>
    import somepkg
E   ModuleNotFoundError: No module named 'somepkg'
============================================== short test summary info ==============================================
ERROR tests/test_foo.py
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
================================================= 1 error in 0.06s ==================================================

Case 1: python -m pytest

This is the solution from pytest docs. I added "python.testing.pytestPath": "python -m pytest" to .vscode/settings.json and tried to launch tests.

VS Code, however, still fails on discovery.

> /usr/bin/python ~/.vscode/extensions/ms-python.python-2020.12.424452561/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir ~/Documents/VSCode-Pytest-Bug -s --cache-clear .
cwd: ~/Documents/VSCode-Pytest-Bug
Error 2020-12-27 19:55:08: Failed to parse discovered Test [r [Error]: ============================= test session starts ==============================
platform linux -- Python 3.8.6, pytest-6.2.1, py-1.9.0, pluggy-0.13.1
rootdir: /home/mikolaj/Documents/VSCode-Pytest-Bug
collected 0 items / 1 error

==================================== ERRORS ====================================
______________________ ERROR collecting tests/test_foo.py ______________________
ImportError while importing test module '/home/mikolaj/Documents/VSCode-Pytest-Bug/tests/test_foo.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib/python3.8/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/test_foo.py:1: in <module>
    import somepkg
E   ModuleNotFoundError: No module named 'somepkg'
=========================== short test summary info ============================
ERROR tests/test_foo.py
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
===================== no tests collected, 1 error in 0.06s =====================

Traceback (most recent call last):
  File "/home/mikolaj/.vscode/extensions/ms-python.python-2020.12.424452561/pythonFiles/testing_tools/run_adapter.py", line 22, in <module>
    main(tool, cmd, subargs, toolargs)
  File "/home/mikolaj/.vscode/extensions/ms-python.python-2020.12.424452561/pythonFiles/testing_tools/adapter/__main__.py", line 100, in main
    parents, result = run(toolargs, **subargs)
  File "/home/mikolaj/.vscode/extensions/ms-python.python-2020.12.424452561/pythonFiles/testing_tools/adapter/pytest/_discovery.py", line 44, in discover
    raise Exception("pytest discovery failed (exit code {})".format(ec))
Exception: pytest discovery failed (exit code 2)

	at ChildProcess.<anonymous> (/home/mikolaj/.vscode/extensions/ms-python.python-2020.12.424452561/out/client/extension.js:9:567859)
	at Object.onceWrapper (events.js:313:26)
	at ChildProcess.emit (events.js:223:5)
	at maybeClose (internal/child_process.js:1021:16)
	at Socket.<anonymous> (internal/child_process.js:430:11)
	at Socket.emit (events.js:223:5)
	at Pipe.<anonymous> (net.js:664:12)]

And running pytest manually works fine.

$ python -m pytest
================================================ test session starts ================================================
platform linux -- Python 3.8.6, pytest-6.2.1, py-1.9.0, pluggy-0.13.1
rootdir: /home/mikolaj/Documents/VSCode-Pytest-Bug
collected 1 item                                                                                                    

tests/test_foo.py .                                                                                           [100%]

================================================= 1 passed in 0.01s =================================================

Case 2: Absolute PYTHONPATH

Here, two new keys were added to the original settings.json: "python.envFile": "${workspaceFolder}/.vscode/.env" and "terminal.integrated.env.linux": { "PYTHONPATH": "${workspaceFolder}" }; and the .vscode/.env file was created with the following content. The path is hard-coded to circumvent issue #13749.

# Somehow using ${workspaceFolder} or ${env:PWD} doesn't work here
PYTHONPATH = /home/mikolaj/Documents/VSCode-Pytest-Bug/

And now test discovery works! But then, running the tests errors out.

> /usr/bin/python ~/.vscode/extensions/ms-python.python-2020.12.424452561/pythonFiles/pyvsc-run-isolated.py pytest --override-ini junit_family=xunit1 --rootdir ~/Documents/VSCode-Pytest-Bug --junit-xml=/tmp/tmp-12233yWmGcDH2XD1w.xml .
cwd: ~/Documents/VSCode-Pytest-Bug

============================= test session starts ==============================
platform linux -- Python 3.8.6, pytest-6.2.1, py-1.9.0, pluggy-0.13.1
rootdir: /home/mikolaj/Documents/VSCode-Pytest-Bug
collected 0 items / 1 error

==================================== ERRORS ====================================
______________________ ERROR collecting tests/test_foo.py ______________________
ImportError while importing test module '/home/mikolaj/Documents/VSCode-Pytest-Bug/tests/test_foo.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib/python3.8/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/test_foo.py:1: in <module>
    import somepkg
E   ModuleNotFoundError: No module named 'somepkg'
-------------- generated xml file: /tmp/tmp-10236qdrGjj8MkHVf.xml --------------
=========================== short test summary info ============================
ERROR tests/test_foo.py
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
=============================== 1 error in 0.07s ===============================

And of course, pytest works like a charm:

$ PYTHONPATH=/home/mikolaj/Documents/VSCode-Pytest-Bug/ pytest
================================================ test session starts ================================================
platform linux -- Python 3.8.6, pytest-6.2.1, py-1.9.0, pluggy-0.13.1
rootdir: /home/mikolaj/Documents/VSCode-Pytest-Bug
collected 1 item                                                                                                    

tests/test_foo.py .                                                                                           [100%]

================================================= 1 passed in 0.01s =================================================

Case 3: relative PYTHONPATH

This time settings.json looks like this:

{
    "python.testing.pytestArgs": [
        "."
    ],
    "python.testing.unittestEnabled": false,
    "python.testing.nosetestsEnabled": false,
    "python.testing.pytestEnabled": true,
    "python.testing.pytestPath": "python -m pytest",
    "python.envFile": "${workspaceFolder}/.vscode/.env",
    "terminal.integrated.env.linux": {
        "PYTHONPATH": "."
    },
    "python.testing.cwd": "${workspaceFolder}"
}

And .vscode/.env contains only PYTHONPATH=.

VS Code now correctly discovers the tests, but crashes when trying to run them:

> /usr/bin/python ~/.vscode/extensions/ms-python.python-2020.12.424452561/pythonFiles/pyvsc-run-isolated.py pytest --override-ini junit_family=xunit1 --rootdir ~/Documents/VSCode-Pytest-Bug --junit-xml=/tmp/tmp-10236MyRbp3P5xcEN.xml .
cwd: ~/Documents/VSCode-Pytest-Bug

============================= test session starts ==============================
platform linux -- Python 3.8.6, pytest-6.2.1, py-1.9.0, pluggy-0.13.1
rootdir: /home/mikolaj/Documents/VSCode-Pytest-Bug
collected 0 items / 1 error

==================================== ERRORS ====================================
______________________ ERROR collecting tests/test_foo.py ______________________
ImportError while importing test module '/home/mikolaj/Documents/VSCode-Pytest-Bug/tests/test_foo.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib/python3.8/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/test_foo.py:1: in <module>
    import somepkg
E   ModuleNotFoundError: No module named 'somepkg'
-------------- generated xml file: /tmp/tmp-10236whE7Tokmzock.xml --------------
=========================== short test summary info ============================
ERROR tests/test_foo.py
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
=============================== 1 error in 0.07s ===============================

And, yet again, pytest works fine:

$ PYTHONPATH=. pytest                                         
================================================ test session starts ================================================
platform linux -- Python 3.8.6, pytest-6.2.1, py-1.9.0, pluggy-0.13.1
rootdir: /home/mikolaj/Documents/VSCode-Pytest-Bug
collected 1 item                                                                                                    

tests/test_foo.py .                                                                                           [100%]

================================================= 1 passed in 0.01s =================================================

Expected behavior

I would expect that modifications from cases 1, 2 and 3 allow testing non-installed modules.

Related issues

#6891 - that's not the case. starting from case 3, I have created a separate test in tests/test_env.py:

import os

def test_pythonpath():
    assert os.getenv("PYTHONPATH") == "."

And it passes correctly.

#12420 - there "Run tests" work, here it doesn't

[kimadeline]

Hi @MKuranowski, thank you for reaching out and your thorough research!

I can repro your issue, this seems to be a regression (it used to work), marking it as such.

Thank you again for the report!

[nss350]

Hi,
I have encountered a similar issue on Windows. I'm developing a package with a directory structure similar to the one described in the original issue. Additionally, I am using a virtual environment and test discovery appears to work without much issue after adding the following:

"python.pythonPath": /path/to/my/venv/python
"python.testing.pytestEnabled": true

Further, adding this setting allows py.test tests to run from command line. Note, pytest is installed in my virtual environment.

"terminal.integrated.env.windows": {
        "PYTHONPATH": "${PYTHONPATH};${workspaceFolder}"
}

However, I am unable to get the tests to run successfully using the testing interface in vs code. They all fail with import errors. I have gone round and round on this, but nothing seems to work. I initially tried adding a .venv file pointing PYTHONPATH to the current working folder which did not work. Next, I set the following settings:

python.analysis.extraPaths
python.autoComplete.extraPaths (I now believe this is deprecated)

Neither of which worked.

Finally, I made a second copy of my project and pointed my .env file to the copied project folder and finally it worked. Therefore, it does seem that the .env is picked up, but there is no way to add the current working directory to the PYTHONPATH or that this is somehow overwritten.

If it helps, I can provide a more complete example, but it would be quite similar to the one posted here. Note that the Python Text Explorer extension appears to find everything appropriately.

Is there any update on this issue?

[kimadeline]

Hi @nss350,

No progress on this issue yet, feel free to 👍 it to help bumping up its priority.

Thanks!

[jocelyne8]

I believe I have a related, but not identical issue. I have been downgrading to extension version v2020.9.111407 (or so) every time it auto-updates to get around it. In my case, the package is installed with pip install -e , so there is an egg-link in the package directory. VS code discovers the tests, the tests complete in debug mode, but give an import error for the package when I run them.

[luabud]

Hey @MKuranowski, can you try adding a __init__.py file to your tests folder and see if it works?

[MKuranowski]

After some time I also found that workaround, so yes, adding tests/__init__.py fixes the issue. Another workaround is to add an empty conftest.py to project root.

[luabud]

Thanks for checking!

[SuaveFool]

Thanks @luabud I was having the same issue (exactly as described, but in Windows), adding the __init__.py to my tests folder fixes it for me.

[jaron-l]

Forgive me if I don't understand, but is this because pytest run is not obeying the environment variables set in .vscode/.env when running the test? It seems case 3 seems to point to that. This seems related to how PYTHONPATH is set.

[brettcannon]

@jaron-l I don't think so; "terminal.integrated.env.linux" isn't used by us as we are not running pytest through the terminal.

[jaron-l]

@brettcannon What about "python.envFile"? Not being able to find packages is a python path issue so the different scenarios have different python paths.

[brettcannon]

@jaron-l if you put the tests/ directory on to sys.path it might make it work (although that also depends on the tests not importing things in such a way that it won't break from that happening).

[phloose]

I am having the same issue with an editable install (pip install -e). To make the extension run the tests i have to add an __init__.py as @luabud has suggested and others have also successfully tried. Tests discovery works fine without.

I am just wondering why i can easily run pytest from the command line WITHOUT adding an __init__.py in the tests folder. Personally i don't want to clutter the test folder with unnecessary files only to make the extension being able to run the tests.

How does the extension invoke pytest?

[jocelyne8]

It sounds like it could be the same as this issue: #14579
which is fixed in the dev build.
My solution has been to install v2020.9.114305 every time it upgrades the package.

[phloose]

As suggested in #14570 (comment) using the user setting "python.useIsolation": false fixes the issue for me.

[Asaurus1]

Also having this issue in Version 1.55.0 on Windows x64 and python 3.7.

• Running python in a virtual env

• Have installed a model "dalek" in my venv using pip install -e dalek (shows up on pip list) | VSCode testing framework reports "no module named 'dalek'

• Have a folder 'borg' in my project directory with an __init__.py file in it so it can be imported with import borg. | VSCode testing framework reports "no module named 'borg'"
  

• Adding an __init__.py file to my /tests directory as suggested by @luabud DOES seem to work for me.

[EliiseS]

I'm using a devcontainer and I'm having the same issue with my modules. Our tests on the command line also use python -m pytest to run the tests. Adding conftest.py to root of the workspace or __init__.py to tests\ has not worked either for me.

I got the test discovery to work as well by adding "containerEnv": {"PYTHONPATH": "${containerWorkspaceFolder}"}, to the .devcontainer/.devcontainer.json. The Run Test highlight still fails, however, on a very surprising note, debug test does work for running the tests.

[heitorlessa]

I wrestled with this today and ended up having a .env file with PYTHONPATH=.. This ensured VSCode updated PYTHONPATH to include the current ${workspaceFolder} correctly - just reloaded and worked like a charm.

[eleanorjboyd]

Hello! I have reviewed this issue and given the architectural changes of the testing rewrite, this issue should no longer exist on the rewrite. Therefore I am going to close this issue but please comment or open a new issue if you are still seeing a problem when you try this yourself!

To use the rewrite yourself just add ”python.experiments.optInto": ["pythonTestAdapter"] to your user settings. You can confirm you have the rewrite enabled by setting your log level to trace, via the Developer: Set Log Level command in the command palette. Then check to see if Experiment 'pythonTestAdapter' is active is in your python logs.

Thanks!