Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Debugging pytest tests ignore the set .env file #12420

Closed
flaker opened this issue Jun 17, 2020 · 9 comments
Closed

Debugging pytest tests ignore the set .env file #12420

flaker opened this issue Jun 17, 2020 · 9 comments
Labels
area-testing bug Issue identified by VS Code Team member as probable bug

Comments

@flaker
Copy link

flaker commented Jun 17, 2020

Environment data

  • VS Code version: 1.46.0
  • Extension version (available under the Extensions sidebar): v2020.6.88468
  • OS and version: macOS Catalina
  • Python version (& distribution if applicable, e.g. Anaconda): Python 3.8.2
  • Type of virtual environment used (N/A | venv | virtualenv | conda | ...): pipenv
  • Relevant/affected Python packages and their versions: pytest 5.4.2
  • Relevant/affected Python-related VS Code extensions and their versions: XXX
  • Jedi or Language Server? (i.e. what is "python.jediEnabled" set to; more info How to update the language server to the latest stable version #3977): Lang server
  • Value of the python.languageServer setting: "Microsoft"

Expected behaviour

Click on "Run Test" will execute the test.
Click on "Debut Test" will execute the test and stop in my breakpoint.

Actual behaviour

"Run Test" works as expected.
"Debug Test" starts the test execution but fails to load a library.

Steps to reproduce:

[NOTE: Self-contained, minimal reproducing code samples are extremely helpful and will expedite addressing your issue]

Create a python project with the following format:

/project/backend/[flask app resides here]
/project/backend/test/[tests for said app resides here]
/project/backend-lib/be_lib/[some basic code used like a library resides here]
/project/backend/.env.local_test    [in this file, `PYTHONPATH` is set to also be able to find `be_lib`]

Set:

"python.envFile": "/Users/myuser/dev/project/backend/.env.local_test"

The content of .env.local_test

# Environment variable overrides for local development
PYTHONPATH=../backend-lib:$PYTHONPATH
FLASK_APP=api.app
FLASK_ENV=development
DEBUG=true

It's important to notice that discovery and (non debug) run are behaving properly.

Logs

From Debug Console:

ImportError while loading conftest '/Users/myuser/dev/project/backend/test/conftest.py'.
test/conftest.py:1: in <module>
    from test import settings as test_app_settings
test/settings.py:1: in <module>
    from api.settings import *  # NOQA
api/settings/__init__.py:1: in <module>
    from .app import *  # NOQA
api/settings/app.py:10: in <module>
    from be_lib.connexion import resolver as lib_resolver
E   ModuleNotFoundError: No module named 'be_lib'

There is no output from under "Python" or "Python Test Logs"

@flaker flaker added triage-needed Needs assignment to the proper sub-team bug Issue identified by VS Code Team member as probable bug labels Jun 17, 2020
@ghost ghost removed the triage-needed Needs assignment to the proper sub-team label Jun 18, 2020
@ericsnowcurrently
Copy link
Member

@flaker, thanks for letting us know about this. We'll take a look and get back to you as soon as possible.

@ericsnowcurrently
Copy link
Member

The "python.envFile" setting definitely should be respected when debugging tests. The relevant extension code uses that setting by default when launching the debugger for tests.

You can also set "envFile" in your debugger config (in .vscode/launch.json) which overrides the workspace config. Likewise you can use the "env" setting in your debugger config to set environment variables directly. However, those debugger config settings are only respected for tests if the relevant config in launch.json has "request": "test".

(See: https://code.visualstudio.com/docs/python/testing#_debug-tests)

Do you get the same problem if you add a debugger config with "envFile" set in it, like the following, in .vscode/launch.json:

{
    "name": "Debug Flask Tests",
    "type": "python",
    "request": "test",
    "envFile": "/Users/myuser/dev/project/backend/.env.local_test"
}

What about if you set "env" there?

{
    "name": "Debug Flask Tests",
    "type": "python",
    "request": "test",
    "env": {
        "PYTHONPATH": "../backend-lib:$PYTHONPATH",
        "FLASK_APP": "api.app",
        "FLASK_ENV": "development",
        "DEBUG": true
    }
}

Using the explicit "env" will help to identify if the debugger is having trouble with the content of the .env file.

Also, please refer to the docs on debugging flask apps, though that doesn't relate to running tests.

@ericsnowcurrently ericsnowcurrently added the info-needed Issue requires more information from poster label Jun 22, 2020
@flaker
Copy link
Author

flaker commented Jun 22, 2020

Hello @ericsnowcurrently
Thanks for the quick answer. I am puzzled as to why "run" and discovery works and not "debug". So, testing your suggestions with and without python.envFile (this broke "run" and discoverability)


Suggestion 1:

(from the path bar)
under backend > .vscode > {} launch.json > Launch Targets ? {} Debug Flask Tests

    "version": "0.2.0",
    "configurations": [
        {
            "name": "Debug Flask Tests",
            "type": "python",
            "request": "test",
            "envFile": "/Users/myuser/dev/project/backend/.env.local_test"
        }
    ]

It did not work, showed the exact same error. The modification to PYTHONPATH inside .env.local_test never took place. Output:

ImportError while loading conftest '/Users/myuser/dev/project/backend/test/conftest.py'.
test/conftest.py:1: in <module>
    from test import settings as test_app_settings
test/settings.py:1: in <module>
    from api.settings import *  # NOQA
api/settings/__init__.py:1: in <module>
    from .app import *  # NOQA
api/settings/app.py:10: in <module>
    from be_lib.connexion import resolver as lib_resolver
E   ModuleNotFoundError: No module named 'be_lib'

Suggestion 2:

    "version": "0.2.0",
    "configurations": [
        {
            "name": "Debug Flask Tests",
            "type": "python",
            "request": "test",
            "env": {
                "PYTHONPATH": "../backend-lib:$PYTHONPATH",
                "FLASK_APP": "api.app",
                "FLASK_ENV": "development",
                "DEBUG": "true",
            }
        }
    ]

failed with the exact same error. I've also tried using a full PYTHONPATH on that last solution. Thinking that the extension might be having issues with the relative path.

Let me know if you want to try something else.

@ericsnowcurrently
Copy link
Member

Thanks, that helps. It looks like our handling of .env files isn't quite right in the testing case (but correct in the other debug cases). In fact, the code for the debug config used for testing is almost completely distinct from that used for other debug scenarios. The same goes, to a lesser extent, for other uses of .env files in the Python extension. From what I can tell, we don't do any substitution relative to the process environment variables.

Regardless, we will investigate further and work on a fix.

@ericsnowcurrently ericsnowcurrently added area-testing needs PR and removed info-needed Issue requires more information from poster triage labels Jun 23, 2020
@ericsnowcurrently ericsnowcurrently removed their assignment Jun 23, 2020
@flaker
Copy link
Author

flaker commented Jun 29, 2020

Leaving this comment in case someone is also bumping into this.

So, I was trying to workaround this and thought "I can use a pth file". Added the file to the pipenv created venv site-packages (pipenv run python -c "import be_lib" works fine). With debugging tests, no luck.

Then I thought. Maybe it's pointing to the wrong pytest ignoring the pipenv-ed one, so I pointed directly to the pyenv that was supposed to be used. Still no luck.

@rajpdus
Copy link

rajpdus commented Jul 14, 2020

Hello @flaker

I concur. The issue exists and have been trying to wrap my head around trying various other tips and tricks. I tried additionally doing the following with no luck:

  • Setup a global PYTHONPATH to point to the additional package dir.
  • Setup PYTHONPATH in terminal.integrated.env.osx
  • .env file is not respected.

Additionally I check sys.path does not contain PYTHONPATH available in os.environ

@Laibalion
Copy link

I'm having spent almost a day trying to figure out why my tests don't debug, only to just now find out it's a VS Code bug :|.
Please fix this, as there seemingly is no workaround

@flaker
Copy link
Author

flaker commented Aug 6, 2020

I'm having spent almost a day trying to figure out why my tests don't debug, only to just now find out it's a VS Code bug :|.
Please fix this, as there seemingly is no workaround

For the moment what I did is just be sure my tests have the proper sys.path in my conftest file (adding the missing paths if required). That is good enough while waiting for a good fix.

@eleanorjboyd
Copy link
Member

Hello! I have reviewed this issue and have tested a similar scenario successfully on our testing rewrite. Therefore I am going to close this issue but please comment or open a new issue if you are still seeing a problem when you try this yourself!

To use the rewrite yourself just add ”python.experiments.optInto": ["pythonTestAdapter"] to your user settings. You can confirm you have the rewrite enabled by setting your log level to trace, via the Developer: Set Log Level command in the command palette. Then check to see if Experiment 'pythonTestAdapter' is active is in your python logs.

Thanks!

@github-actions github-actions bot removed the needs PR Ready to be worked on label Oct 30, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Nov 30, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
area-testing bug Issue identified by VS Code Team member as probable bug
Projects
None yet
Development

No branches or pull requests

7 participants