Debugging pytest tests ignore the set .env file

[flaker]

Environment data

• VS Code version: 1.46.0
• Extension version (available under the Extensions sidebar): v2020.6.88468
• OS and version: macOS Catalina
• Python version (& distribution if applicable, e.g. Anaconda): Python 3.8.2
• Type of virtual environment used (N/A | venv | virtualenv | conda | ...): pipenv
• Relevant/affected Python packages and their versions: pytest 5.4.2
• Relevant/affected Python-related VS Code extensions and their versions: XXX
• Jedi or Language Server? (i.e. what is "python.jediEnabled" set to; more info How to update the language server to the latest stable version #3977): Lang server
• Value of the python.languageServer setting: "Microsoft"

Expected behaviour

Click on "Run Test" will execute the test.
Click on "Debut Test" will execute the test and stop in my breakpoint.

Actual behaviour

"Run Test" works as expected.
"Debug Test" starts the test execution but fails to load a library.

Steps to reproduce:

[NOTE: Self-contained, minimal reproducing code samples are extremely helpful and will expedite addressing your issue]

Create a python project with the following format:

/project/backend/[flask app resides here]
/project/backend/test/[tests for said app resides here]
/project/backend-lib/be_lib/[some basic code used like a library resides here]
/project/backend/.env.local_test    [in this file, `PYTHONPATH` is set to also be able to find `be_lib`]

Set:

"python.envFile": "/Users/myuser/dev/project/backend/.env.local_test"

The content of .env.local_test

# Environment variable overrides for local development
PYTHONPATH=../backend-lib:$PYTHONPATH
FLASK_APP=api.app
FLASK_ENV=development
DEBUG=true

It's important to notice that discovery and (non debug) run are behaving properly.

Logs

From Debug Console:

ImportError while loading conftest '/Users/myuser/dev/project/backend/test/conftest.py'.
test/conftest.py:1: in <module>
    from test import settings as test_app_settings
test/settings.py:1: in <module>
    from api.settings import *  # NOQA
api/settings/__init__.py:1: in <module>
    from .app import *  # NOQA
api/settings/app.py:10: in <module>
    from be_lib.connexion import resolver as lib_resolver
E   ModuleNotFoundError: No module named 'be_lib'

There is no output from under "Python" or "Python Test Logs"

[ericsnowcurrently]

@flaker, thanks for letting us know about this. We'll take a look and get back to you as soon as possible.

[ericsnowcurrently]

The "python.envFile" setting definitely should be respected when debugging tests. The relevant extension code uses that setting by default when launching the debugger for tests.

You can also set "envFile" in your debugger config (in .vscode/launch.json) which overrides the workspace config. Likewise you can use the "env" setting in your debugger config to set environment variables directly. However, those debugger config settings are only respected for tests if the relevant config in launch.json has "request": "test".

(See: https://code.visualstudio.com/docs/python/testing#_debug-tests)

Do you get the same problem if you add a debugger config with "envFile" set in it, like the following, in .vscode/launch.json:

{
    "name": "Debug Flask Tests",
    "type": "python",
    "request": "test",
    "envFile": "/Users/myuser/dev/project/backend/.env.local_test"
}

What about if you set "env" there?

{
    "name": "Debug Flask Tests",
    "type": "python",
    "request": "test",
    "env": {
        "PYTHONPATH": "../backend-lib:$PYTHONPATH",
        "FLASK_APP": "api.app",
        "FLASK_ENV": "development",
        "DEBUG": true
    }
}

Using the explicit "env" will help to identify if the debugger is having trouble with the content of the .env file.

Also, please refer to the docs on debugging flask apps, though that doesn't relate to running tests.

[flaker]

Hello @ericsnowcurrently
Thanks for the quick answer. I am puzzled as to why "run" and discovery works and not "debug". So, testing your suggestions with and without python.envFile (this broke "run" and discoverability)

---

Suggestion 1:

(from the path bar)
under backend > .vscode > {} launch.json > Launch Targets ? {} Debug Flask Tests

    "version": "0.2.0",
    "configurations": [
        {
            "name": "Debug Flask Tests",
            "type": "python",
            "request": "test",
            "envFile": "/Users/myuser/dev/project/backend/.env.local_test"
        }
    ]

It did not work, showed the exact same error. The modification to PYTHONPATH inside .env.local_test never took place. Output:

ImportError while loading conftest '/Users/myuser/dev/project/backend/test/conftest.py'.
test/conftest.py:1: in <module>
    from test import settings as test_app_settings
test/settings.py:1: in <module>
    from api.settings import *  # NOQA
api/settings/__init__.py:1: in <module>
    from .app import *  # NOQA
api/settings/app.py:10: in <module>
    from be_lib.connexion import resolver as lib_resolver
E   ModuleNotFoundError: No module named 'be_lib'

---

Suggestion 2:

    "version": "0.2.0",
    "configurations": [
        {
            "name": "Debug Flask Tests",
            "type": "python",
            "request": "test",
            "env": {
                "PYTHONPATH": "../backend-lib:$PYTHONPATH",
                "FLASK_APP": "api.app",
                "FLASK_ENV": "development",
                "DEBUG": "true",
            }
        }
    ]

failed with the exact same error. I've also tried using a full PYTHONPATH on that last solution. Thinking that the extension might be having issues with the relative path.

Let me know if you want to try something else.

[ericsnowcurrently]

Thanks, that helps. It looks like our handling of .env files isn't quite right in the testing case (but correct in the other debug cases). In fact, the code for the debug config used for testing is almost completely distinct from that used for other debug scenarios. The same goes, to a lesser extent, for other uses of .env files in the Python extension. From what I can tell, we don't do any substitution relative to the process environment variables.

Regardless, we will investigate further and work on a fix.

[flaker]

Leaving this comment in case someone is also bumping into this.

So, I was trying to workaround this and thought "I can use a pth file". Added the file to the pipenv created venv site-packages (pipenv run python -c "import be_lib" works fine). With debugging tests, no luck.

Then I thought. Maybe it's pointing to the wrong pytest ignoring the pipenv-ed one, so I pointed directly to the pyenv that was supposed to be used. Still no luck.

[rajpdus]

Hello @flaker

I concur. The issue exists and have been trying to wrap my head around trying various other tips and tricks. I tried additionally doing the following with no luck:

• Setup a global PYTHONPATH to point to the additional package dir.
• Setup PYTHONPATH in terminal.integrated.env.osx
• .env file is not respected.

Additionally I check sys.path does not contain PYTHONPATH available in os.environ

[Laibalion]

I'm having spent almost a day trying to figure out why my tests don't debug, only to just now find out it's a VS Code bug :|.
Please fix this, as there seemingly is no workaround

[flaker]

I'm having spent almost a day trying to figure out why my tests don't debug, only to just now find out it's a VS Code bug :|.
Please fix this, as there seemingly is no workaround

For the moment what I did is just be sure my tests have the proper sys.path in my conftest file (adding the missing paths if required). That is good enough while waiting for a good fix.

[eleanorjboyd]

Hello! I have reviewed this issue and have tested a similar scenario successfully on our testing rewrite. Therefore I am going to close this issue but please comment or open a new issue if you are still seeing a problem when you try this yourself!

To use the rewrite yourself just add ”python.experiments.optInto": ["pythonTestAdapter"] to your user settings. You can confirm you have the rewrite enabled by setting your log level to trace, via the Developer: Set Log Level command in the command palette. Then check to see if Experiment 'pythonTestAdapter' is active is in your python logs.

Thanks!