pytest broken (run fails on module discovery, debug works)

[schperplata]

Issue Type: Bug
With the latest Python extension update (2020.10.332292344), pytests are unable to run tests, while I can debug them normally. This happens only when pytests are run via VS Code. If run via cmd, this is is not the case.
Works on previous version: v2020.9.114305.

I discovered that new plugin has different sys.path list.
Debug (working):

sys.path:
    w:\AsystLib\isys\python
    W:\APPS\PythonModules
    ...

While run (failing):

sys.path:
    c:\Users\domen\.vscode\extensions\ms-python.python-2020.10.332292344\pythonFiles
    W:\APPS\PythonModules

As you can see, first enty: w:\AsystLib\isys\python is missing and is replaced with VSC extension path. Error:

=================================== ERRORS ====================================
___________________ ERROR collecting isys_tests/test_svn.py ___________________
ImportError while importing test module 'w:\AsystLib\isys\python\isys_tests\test_svn.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
isys_tests\test_svn.py:13: in <module>
    import isys.generic
E   ModuleNotFoundError: No module named 'isys'
- generated xml file: C:\Users\domen\AppData\Local\Temp\tmp-22572SU2jr85kY5KB.xml -
=========================== short test summary info ===========================
ERROR isys_tests/test_svn.py
============================== 1 error in 0.03s ===============================
ERROR: not found: w:\AsystLib\isys\python\isys_tests\test_svn.py::test_getStatus
(no name 'w:\\AsystLib\\isys\\python\\isys_tests\\test_svn.py::test_getStatus' in any of [<Module isys_tests/test_svn.py>])

isys is a folder inside: w:\AsystLib\isys\python where many modules are placed, for example isys.generic. Module is accessible in standard run/debug launch configuratoin, but not in pytest run mode.

I couldn't reproduce this with the most simple single-folder VSC workspace. My workspace merge a couple of folders, some with and some without VSC wks/settings files. There are errors in other workspace folders, but this does not explain this scenario, where debug works while run doesn't.

Let me know how can I help since this is again, major drawback in workflow.

Extension version: 2020.10.332292344
VS Code version: Code 1.50.1 (d2e414d9e4239a252d1ab117bd7067f125afd80a, 2020-10-13T15:06:15.712Z)
OS version: Windows_NT x64 10.0.18363

System Info

Item	Value
CPUs	Intel(R) Core(TM) i7-6820HQ CPU @ 2.70GHz (8 x 2712)
GPU Status	2d_canvas: enabled flash_3d: enabled flash_stage3d: enabled flash_stage3d_baseline: enabled gpu_compositing: enabled multiple_raster_threads: enabled_on oop_rasterization: disabled_off opengl: enabled_on protected_video_decode: unavailable_off rasterization: enabled skia_renderer: disabled_off_ok video_decode: enabled vulkan: disabled_off webgl: enabled webgl2: enabled
Load (avg)	undefined
Memory (System)	15.92GB (6.63GB free)
Process Argv	W:\AsystLib\isys\python\VSCodeDevWorkspace_refactor.code-workspace --crash-reporter-id 919e0ed9-46f3-4ad4-9e28-f35755bf8266
Screen Reader	no
VM	0%

[jamesstidard]

I'm seeing the same.

FYI @schperplata, if it's interrupting your workflow, you can install the previous version:

[akukuq]

This is likely due to the changes in pyvsc-run-isolated.py script. See related issue #14570.

[schperplata]

Just a clarification, I don't use pip 'editable' mode. My folders with modules are added to %PYTHONPATH% process env var prior VSCode is run

[s6mike]

I had the same problem on Linux version of VSC - it just started today. Problems panel and pytest both reported not finding my module (which I install in pip editable mode). But I could run my tests from the terminal with pytest and the code otherwise ran fine. And the tests would also run in debug mode.

I just rolled back to version 2020.9.114305 and it now works fine.

I also had this sys path added when running the test but not otherwise:

.vscode/extensions/ms-python.python-2020.10.332292344/pythonFiles

[mrdavidlaing]

I'm seeing the same issue on Mac - pytest and pylint failing after #14570 removed the PYTHONPATH=. setting in my ${workspaceFolder}/.env file.

Rolling back to version 2020.9.114305 solved the problem for me.

[karrtikr]

Just want to quickly knowledge that we've received the issue and will be getting back to you soon!

[karrtikr]

@schperplata Unfortunately, I am unable to reproduce this issue with a simple folder. My test file looked like this,

And my .env file had PYTHONPATH=./src. It printed sys.path as expected.

['c:\\Users\\karraj\\OneDrive - Microsoft\\Desktop\\folders\\memeplex', 'c:\\Users\\karraj\\.vscode\\extensions\\ms-python.python-2020.10.332292344\\pythonFiles', 'c:\\Users\\karraj\\OneDrive - Microsoft\\Desktop\\folders\\memeplex\\src'.....]

• Can you please give me minimal setup where you can repro this?
• Also, please provide the logs as mentioned in the issue template which should contain the command extension runs when you run tests.

Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)

XXX

P.S it shouldn't replace anything on your sys.path. The current setup is expected to only removes '' and cwd from your sys.path. #14014

[schperplata]

Here is a simpe example that reproduce exactly the same outcome: works in 2020.9.114305, and in debug mode on the latest version, but not in run mode.

pytestExample.zip

Python Test log tab:

python c:\Users\domen\.vscode\extensions\ms-python.python-2020.10.332292344\pythonFiles\testing_tools\run_adapter.py discover pytest -- --rootdir c:\Users\domen\Desktop\tmp\pytestExample -s --cache-clear tests
============================= test session starts =============================
platform win32 -- Python 3.8.2, pytest-5.4.1, py-1.8.1, pluggy-0.13.1
rootdir: c:\Users\domen\Desktop\tmp\pytestExample
collected 0 items / 1 error

=================================== ERRORS ====================================
___________________ ERROR collecting tests/test_example.py ____________________
ImportError while importing test module 'C:\Users\domen\Desktop\tmp\pytestExample\tests\test_example.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
C:\Users\domen\Desktop\tmp\tests\test_example.py:1: in <module>
    ???
E   ModuleNotFoundError: No module named 'src1'
- generated xml file: C:\Users\domen\AppData\Local\Temp\tmp-310564dQFuCDpC6FV.xml -
=========================== short test summary info ===========================
ERROR tests/test_example.py
!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
============================== 1 error in 0.12s ===============================

Python tab:

> C:\Python3.8.2\python.exe c:\Users\domen\.vscode\extensions\ms-python.python-2020.10.332292344\pythonFiles\pyvsc-run-isolated.py pytest --override-ini junit_family=xunit1 --rootdir c:\Users\domen\Desktop\tmp\pytestExample --junit-xml=C:\Users\domen\AppData\Local\Temp\tmp-310564dQFuCDpC6FV.xml ./tests/test_example.py
cwd: c:\Users\domen\Desktop\tmp\pytestExample

The behaviour is the same with Python 3.6 (originally reported).

The current setup is expected to only removes '' and cwd from your sys.path.

This is probably the cause?

[mrdavidlaing]

Here is a sample app that reproduces the problem for me - https://github.com/mrdavidlaing/vscode-python-issue14579-example.

Here is a screenshot showing the errors I see when using 2020.10.332292344

• pylint is failing on line 3: from awesome_pkg import thing_doer
• pytest is failing on the same import line: E ModuleNotFoundError: No module named 'awesome_pkg'

Output from the Output > Python panel is as follows:

----------Generating Tags----------
ctags --options=/Users/dlaing/.vscode/extensions/ms-python.python-2020.10.332292344/resources/ctagOptions --languages=Python --exclude=**/site-packages/** -o /Users/dlaing/workspace/vscode-python-issue14579-example/.vscode/tags .
> ctags --options=/Users/dlaing/.vscode/extensions/ms-python.python-2020.10.332292344/resources/ctagOptions --languages=Python --exclude=**/site-packages/** -o ~/workspace/vscode-python-issue14579-example/.vscode/tags .
cwd: ~/workspace/vscode-python-issue14579-example
ctags: Warning: --extra option is obsolete; use --extras instead
> ~/.local/share/virtualenvs/vscode-python-issue14579-example-j6G2tqz-/bin/python ~/.vscode/extensions/ms-python.python-2020.10.332292344/pythonFiles/pyvsc-run-isolated.py pylint --disable=all --enable=F,unreachable,duplicate-key,unnecessary-semicolon,global-variable-not-assigned,unused-variable,unused-wildcard-import,binary-op-exception,bad-format-string,anomalous-backslash-in-string,bad-open-mode,E0001,E0011,E0012,E0100,E0101,E0102,E0103,E0104,E0105,E0107,E0108,E0110,E0111,E0112,E0113,E0114,E0115,E0116,E0117,E0118,E0202,E0203,E0211,E0213,E0236,E0237,E0238,E0239,E0240,E0241,E0301,E0302,E0303,E0401,E0402,E0601,E0602,E0603,E0604,E0611,E0632,E0633,E0701,E0702,E0703,E0704,E0710,E0711,E0712,E1003,E1101,E1102,E1111,E1120,E1121,E1123,E1124,E1125,E1126,E1127,E1128,E1129,E1130,E1131,E1132,E1133,E1134,E1135,E1136,E1137,E1138,E1139,E1200,E1201,E1205,E1206,E1300,E1301,E1302,E1303,E1304,E1305,E1306,E1310,E1700,E1701 --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text ~/workspace/vscode-python-issue14579-example/awesome_pkg/test/test_thing_doer.py
cwd: ~/workspace/vscode-python-issue14579-example
> ~/.local/share/virtualenvs/vscode-python-issue14579-example-j6G2tqz-/bin/python ~/.vscode/extensions/ms-python.python-2020.10.332292344/pythonFiles/pyvsc-run-isolated.py pylint --disable=all --enable=F,unreachable,duplicate-key,unnecessary-semicolon,global-variable-not-assigned,unused-variable,unused-wildcard-import,binary-op-exception,bad-format-string,anomalous-backslash-in-string,bad-open-mode,E0001,E0011,E0012,E0100,E0101,E0102,E0103,E0104,E0105,E0107,E0108,E0110,E0111,E0112,E0113,E0114,E0115,E0116,E0117,E0118,E0202,E0203,E0211,E0213,E0236,E0237,E0238,E0239,E0240,E0241,E0301,E0302,E0303,E0401,E0402,E0601,E0602,E0603,E0604,E0611,E0632,E0633,E0701,E0702,E0703,E0704,E0710,E0711,E0712,E1003,E1101,E1102,E1111,E1120,E1121,E1123,E1124,E1125,E1126,E1127,E1128,E1129,E1130,E1131,E1132,E1133,E1134,E1135,E1136,E1137,E1138,E1139,E1200,E1201,E1205,E1206,E1300,E1301,E1302,E1303,E1304,E1305,E1306,E1310,E1700,E1701 --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text ~/workspace/vscode-python-issue14579-example/awesome_pkg/test/test_thing_doer.py
cwd: ~/workspace/vscode-python-issue14579-example
##########Linting Output - pylint##########
************* Module test_thing_doer
3,0,error,import-error:Unable to import 'awesome_pkg'

--------------------------------------------------------------------
Your code has been rated at -2.50/10 (previous run: -2.50/10, +0.00)

> ~/.local/share/virtualenvs/vscode-python-issue14579-example-j6G2tqz-/bin/python ~/.vscode/extensions/ms-python.python-2020.10.332292344/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir ~/workspace/vscode-python-issue14579-example -s --cache-clear -o junit_family=xunit1 .
cwd: ~/workspace/vscode-python-issue14579-example

And the Python Test Log

============================= test session starts ==============================
platform darwin -- Python 3.8.6, pytest-6.1.2, py-1.9.0, pluggy-0.13.1
rootdir: /Users/dlaing/workspace/vscode-python-issue14579-example
collected 1 item

awesome_pkg/test/test_syspath.py ['/Users/dlaing/workspace/vscode-python-issue14579-example/awesome_pkg/test', '/Users/dlaing/.vscode/extensions/ms-python.python-2020.10.332292344/pythonFiles', '/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python38.zip', '/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8', '/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/lib-dynload', '/Users/dlaing/.local/share/virtualenvs/vscode-python-issue14579-example-j6G2tqz-/lib/python3.8/site-packages']
.

- generated xml file: /var/folders/cx/c_41bpzs1jlc465fx_k_9qfh0000gn/T/tmp-75980YJBGpPLpDxBW.xml -
============================== 1 passed in 0.02s ===============================
============================= test session starts ==============================
platform darwin -- Python 3.8.6, pytest-6.1.2, py-1.9.0, pluggy-0.13.1
rootdir: /Users/dlaing/workspace/vscode-python-issue14579-example
collected 0 items / 1 error

==================================== ERRORS ====================================
_____________ ERROR collecting awesome_pkg/test/test_thing_doer.py _____________
ImportError while importing test module '/Users/dlaing/workspace/vscode-python-issue14579-example/awesome_pkg/test/test_thing_doer.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
awesome_pkg/test/test_thing_doer.py:3: in <module>
    from awesome_pkg import thing_doer
E   ModuleNotFoundError: No module named 'awesome_pkg'
- generated xml file: /var/folders/cx/c_41bpzs1jlc465fx_k_9qfh0000gn/T/tmp-75980qb4RBaZUAhyE.xml -
=========================== short test summary info ============================
ERROR awesome_pkg/test/test_thing_doer.py
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
=============================== 1 error in 0.10s ===============================
python /Users/dlaing/.vscode/extensions/ms-python.python-2020.10.332292344/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir /Users/dlaing/workspace/vscode-python-issue14579-example -s --cache-clear -o junit_family=xunit1 .

If I downgrade to `2020.9.114305` both these errors go away:

cwd: ~/workspace/vscode-python-issue14579-example
> ~/.local/share/virtualenvs/vscode-python-issue14579-example-j6G2tqz-/bin/python ~/.vscode/extensions/ms-python.python-2020.9.114305/pythonFiles/pyvsc-run-isolated.py pylint --disable=all --enable=F,unreachable,duplicate-key,unnecessary-semicolon,global-variable-not-assigned,unused-variable,unused-wildcard-import,binary-op-exception,bad-format-string,anomalous-backslash-in-string,bad-open-mode,E0001,E0011,E0012,E0100,E0101,E0102,E0103,E0104,E0105,E0107,E0108,E0110,E0111,E0112,E0113,E0114,E0115,E0116,E0117,E0118,E0202,E0203,E0211,E0213,E0236,E0237,E0238,E0239,E0240,E0241,E0301,E0302,E0303,E0401,E0402,E0601,E0602,E0603,E0604,E0611,E0632,E0633,E0701,E0702,E0703,E0704,E0710,E0711,E0712,E1003,E1101,E1102,E1111,E1120,E1121,E1123,E1124,E1125,E1126,E1127,E1128,E1129,E1130,E1131,E1132,E1133,E1134,E1135,E1136,E1137,E1138,E1139,E1200,E1201,E1205,E1206,E1300,E1301,E1302,E1303,E1304,E1305,E1306,E1310,E1700,E1701 --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text ~/workspace/vscode-python-issue14579-example/awesome_pkg/test/test_thing_doer.py
cwd: ~/workspace/vscode-python-issue14579-example
##########Linting Output - pylint##########

--------------------------------------------------------------------
Your code has been rated at 10.00/10 (previous run: 10.00/10, +0.00)

> ~/.local/share/virtualenvs/vscode-python-issue14579-example-j6G2tqz-/bin/python ~/.vscode/extensions/ms-python.python-2020.9.114305/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir ~/workspace/vscode-python-issue14579-example -s --cache-clear -o junit_family=xunit1 .
cwd: ~/workspace/vscode-python-issue14579-example

python /Users/dlaing/.vscode/extensions/ms-python.python-2020.9.114305/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir /Users/dlaing/workspace/vscode-python-issue14579-example -s --cache-clear -o junit_family=xunit1 .
============================= test session starts ==============================
platform darwin -- Python 3.8.6, pytest-6.1.2, py-1.9.0, pluggy-0.13.1
rootdir: /Users/dlaing/workspace/vscode-python-issue14579-example
collected 1 item

awesome_pkg/test/test_syspath.py ['/Users/dlaing/workspace/vscode-python-issue14579-example/awesome_pkg/test', '/Users/dlaing/.vscode/extensions/ms-python.python-2020.9.114305/pythonFiles/.does-not-exist', '/Users/dlaing/workspace/vscode-python-issue14579-example', '/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python38.zip', '/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8', '/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/lib-dynload', '/Users/dlaing/.local/share/virtualenvs/vscode-python-issue14579-example-j6G2tqz-/lib/python3.8/site-packages']
.

- generated xml file: /var/folders/cx/c_41bpzs1jlc465fx_k_9qfh0000gn/T/tmp-76962fN0Z8DyZCHyw.xml -
============================== 1 passed in 0.02s ===============================
============================= test session starts ==============================
platform darwin -- Python 3.8.6, pytest-6.1.2, py-1.9.0, pluggy-0.13.1
rootdir: /Users/dlaing/workspace/vscode-python-issue14579-example
collected 1 item

awesome_pkg/test/test_thing_doer.py .

- generated xml file: /var/folders/cx/c_41bpzs1jlc465fx_k_9qfh0000gn/T/tmp-76962QWZf28ierxOY.xml -
============================== 1 passed in 0.02s ===============================

I believe the key difference is the omission of the "root" folder - /Users/dlaing/workspace/vscode-python-issue14579-example - (as specified by .env containing PYTHONPATH=.) from sys.path under v2020.10.332292344

2020.10.332292344: awesome_pkg/test/test_syspath.py ['/Users/dlaing/workspace/vscode-python-issue14579-example/awesome_pkg/test', '/Users/dlaing/.vscode/extensions/ms-python.python-2020.10.332292344/pythonFiles',                                                                         '/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python38.zip', '/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8', '/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/lib-dynload', '/Users/dlaing/.local/share/virtualenvs/vscode-python-issue14579-example-j6G2tqz-/lib/python3.8/site-packages']
2020.9.114305:     awesome_pkg/test/test_syspath.py ['/Users/dlaing/workspace/vscode-python-issue14579-example/awesome_pkg/test', '/Users/dlaing/.vscode/extensions/ms-python.python-2020.9.114305/pythonFiles/.does-not-exist', '/Users/dlaing/workspace/vscode-python-issue14579-example', '/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python38.zip', '/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8', '/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/lib-dynload', '/Users/dlaing/.local/share/virtualenvs/vscode-python-issue14579-example-j6G2tqz-/lib/python3.8/site-packages']

[CanntAim]

Adding an __init__.py in the test suite folder also seemed to "fix" this particular issue for me without requiring a downgrade. You really shouldn't need it there for the test suite to work though thus the cautious use of the word fix.

[karrtikr]

Thanks all for the info! Should be fixed as part of #14570 which is planned for the next release.

[giuliano-97]

I still have this issue on Ubuntu 20.04 i.e. discovery and debug work, but running all the tests fails because PYTHONPATH is not correctly set.

VSC version: 1.56.2
ms-python.python version: 2021.5.842923320

By rolling back to v2020.9.112786 of ms-python.python this works instead.

[karrtikr]

@giuliano-97 Please open up a new issue and we can help you there.