python.testing.pytestEnabled isn't respected for discovery

[tboddyspargo]

Type: Bug

Behaviour

Expected vs. Actual

Expected: In a workspace with a folder, setting "python.testing.pytestEnabled": false in should prevent pytest discovery from running without impacting the settings of any other folders in the workspace even when .py files are edited and saved in a subdirectory. Correspondingly, when .py files are edited/saved in a workspace folder with "python.testing.pytestEnabled": true, discovery should be triggered only in that workspace folder and with the workspace folder as the cwd.

Actual: Logs are misleading about where discovery is/isn't being run. Editing/saving .py files in a subdirectory of a workspace folder where "python.testing.pytestEnabled": true can unexpectedly trigger discovery with a parent folder (added as a separate workspace folder) as the cwd even when it has explicitly (or implicitly) set "python.testing.pytestEnabled": false instead of the expected workspace folder where it is true.

Steps to reproduce:

1. Construct a workspace like this:

workspace_folder/ (added to workspace)
|--.vscode/
|   |-settings.json (python.testing.pytestEnabled=false)
|--other_folder/ (added to workspace)
|   |--.vscode/
|   |   |-settings.json (python.testing.pytestEnabled=true)
|   |--test_true.py

This should result in test discovery running for both folders (undesirable).

Desired behavior 1: Test discovery should not run from the root workspace_folder, but it should run in other_folder. Log messages should make this obvious (i.e. the following message should not be logged):

2023-10-14 16:47:05.703 [info] Discover tests for workspace name: workspace_folder - uri: /tmp/workspace_folder

Desired behavior 2: Editing workspace_folder/other_folder/test_true.py should not result in test discovery running in a workspace_folder where python.testing.pytestEnabled has been disabled, even if it is an ancestor directory of the file that was edited/saved. Instead, it should run in the context of other_folder where python.testing.pytestEnabled is true.

2023-10-14 16:52:45.852 [info] > ~/.pyenv/versions/3.11.4/bin/python -m pytest -p vscode_pytest --collect-only
2023-10-14 16:52:45.852 [info] cwd: /tmp/workspace_folder

Diagnostic data

• Python version (& distribution if applicable, e.g. Anaconda): 3.8.13
• Type of virtual environment used (e.g. conda, venv, virtualenv, etc.): Pyenv
• Value of the python.languageServer setting: Pylance

Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)

2023-10-13 10:43:45.284 [info] Discover tests for workspace name: workspace_folder - uri: /workspace_folder/.vscode/settings.json
2023-10-13 10:43:46.045 [info] Discover tests for workspace name: workspace_folder - uri: /workspace_folder
2023-10-13 10:45:01.570 [info] Python interpreter path: ./.venv/bin/python
2023-10-13 10:45:04.590 [info] Discover tests for workspace name: other_folder - uri: /workspace_folder/other_folder/tests/REDACTED.py
2023-10-13 10:45:04.590 [info] Running discovery for pytest using the new test adapter.
2023-10-13 10:45:04.597 [info] > pyenv exec python -m pytest -p vscode_pytest
2023-10-13 10:45:04.598 [info] cwd: /workspace_folder

User Settings

Multiroot scenario, following user settings may not apply:

envFile: N/A

languageServer: "Pylance"

testing
• pytestArgs: N/A

experiments
• optInto: ["pythonTestAdapter"]

Extension version: 2023.18.0
VS Code version: Code 1.83.1 (Universal) (f1b07bd25dfad64b0167beb15359ae573aecd2cc, 2023-10-10T23:46:55.789Z)
OS version: Darwin arm64 23.0.0
Modes:

System Info

Item	Value
CPUs	Apple M1 Max (10 x 24)
GPU Status	2d_canvas: enabled canvas_oop_rasterization: enabled_on direct_rendering_display_compositor: disabled_off_ok gpu_compositing: enabled multiple_raster_threads: enabled_on opengl: enabled_on rasterization: enabled raw_draw: disabled_off_ok video_decode: enabled video_encode: enabled vulkan: disabled_off webgl: enabled webgl2: enabled webgpu: enabled
Load (avg)	6, 6, 5
Memory (System)	32.00GB (0.62GB free)
Process Argv	--crash-reporter-id 9c8b74e5-efe2-4756-b68a-91b9fde0fa03
Screen Reader	no
VM	0%

A/B Experiments

vsliv368:30146709
vsreu685:30147344
python383:30185418
vspor879:30202332
vspor708:30202333
vspor363:30204092
vslsvsres303:30308271
vserr242:30382549
pythontb:30283811
vsjup518:30340749
pythonptprofiler:30281270
vshan820:30294714
vstes263:30335439
vscoreces:30445986
vscod805:30301674
binariesv615:30325510
bridge0708:30335490
bridge0723:30353136
vsaa593cf:30376535
pythonvs932:30410667
py29gd2263cf:30856253
vsclangdf:30486550
c4g48928:30535728
dsvsc012cf:30540253
pynewext54:30695312
azure-dev_surveyone:30548225
2e4cg342:30602488
f6dab269:30613381
a9j8j154:30646983
showlangstatbar:30737416
962ge761:30841074
a2ce3375:30757347
pythonfmttext:30731395
fixshowwlkth:30771522
showindicator:30805244
pythongtdpath:30769146
i26e3531:30792625
pythonnosmt12:30797651
pythonidxpt:30805730
pythonnoceb:30805159
copilotsettingt:30859503
synctok:30821570
dsvsc013:30795093
dsvsc014:30804076
diffeditorv2:30821572
dsvsc015:30845448
pythonregdiag:30859691

[tboddyspargo]

Additionally, it seems that saving a test_*.py file in /workspace_folder/other_folder triggers discovery in /workspace_folder, but not in /workspace_folder/other_folder as expected...

[eleanorjboyd]

so to clarify you have the folder workspace_folder open as the workspace? Then for the workspace folder settings there are no testing framework enabled? I am a bit confused about how it is finding tests anyways given a test configuration needs to be set at the workspace root level in order for discovery to occur.

[tboddyspargo]

Thanks for your response, @eleanorjboyd. I tried to refine my reproduction a bit and I identified a contributing factor. If a pyproject.toml file is present in a workspace folder, pytest test discovery will run and pytest.testing.pytestEnabled will be ignored. I believe that's the "root cause" of what I'm experiencing. We have a pyproject.toml file at the workspace root that is meant only to provide some linter/formatter config that can be shared with multiple sub-projects. VSCode seems to be assuming that the presence of that pyproject.toml file should "trump" the settings.json configuration values.

I've uploaded a directory representing a minimal repro of the issue:
workspace_folder.zip Use repro from subsequent comment instead.

In my main repo (not the minimal repro) have even tried replacing the pyproject.toml file with tool-specific configuration files, but VSCode still seems to be assuming that I want pytest discovery to run and ignoring the value of python.testing.pytestEnabled when Python files in any subfolder are edited/saved.

I'm hoping this will help zoom in on the problem area. Is there a place where pytest discovery is "assumed" based on the files found in a workspace folder? Should we modify the logic or order of operations such that python.testing.pytestEnabled still takes precedence?

[tboddyspargo]

Ok, I have some corrections/precisions to make. What has been super confusing is that, regardless of the presence of pyproject.toml or the value of python.testing.pytestEnabled, the following line will be logged on startup/refresh of VSCode when open to the example workspace:

2023-10-14 17:12:37.335 [info] Discover tests for workspace name: workspace_folder - uri: /tmp/workspace_folder

This is misleading. When I have "pytest.testing.pytestEnabled": false, I don't want to discover tests in that workspace. However, I don't see the lines suggesting that vscode_pytest discovery is actually being run... so perhaps the setting is being respected despite the log message.

In contrast, when I save a python file (e.g. workspace_folder/other_folder/test_true.py), pytest discovery will be run (if the logged lines below can be trusted) for an ancestor directory (workspace_folder) despite the fact that "pytest.testing.pytestEnabled": false, is in its settings file.

2023-10-14 17:18:30.059 [info] Discover tests for workspace name: other_folder - uri: /tmp/workspace_folder/other_folder/test_true.py
2023-10-14 17:18:30.059 [info] Running discovery for pytest using the new test adapter.
2023-10-14 17:18:30.062 [info] > ~/.pyenv/versions/3.11.4/bin/python -m pytest -p vscode_pytest --collect-only
2023-10-14 17:18:30.062 [info] cwd: /tmp/workspace_folder
2023-10-14 17:18:30.245 [info] Test server connected to a client.
2023-10-14 17:18:30.246 [info] ResultResolver EOT received for discovery.
2023-10-14 17:18:30.264 [info] Disposing data receiver for /tmp/workspace_folder and deleting UUID; pytest discovery.

As a result, a workspace_folder node will be added to the Test Explorer sidebar. This is undesirable (again, because its settings.json file has "pytest.testing.pytestEnabled": false, in it).

Here is an updated repro workspace: workspace_folder.zip

---

Conclusion:

• I believe Discover tests for workspace name is being prematurely and/or inappropriately logged for workspace folders that have set pytest.testing.pytestEnabled to false (the default).
• I believe that when .py files are edited/saved, triggering test discovery, the incorrect execution/working directory is being identified and the value of that incorrect directory's pytest.testing.pytestEnabled setting is getting ignored even though the value of the correct workspace directory is being respected.
  • In the attached repro, editing/saving test_true.py should ONLY trigger test discovery within cdw: /tmp/workspace_folder/other_folder and other_folder should be the only node in the Test Explorer (and only if its pytest.testing.pytestEnabled value is true).
  • EDIT: It's important to note that this also means discovery may NOT run in the directory you expect it to even when python.testing.pytestEnabled is true (i.e. if you add a new test to test_true.py it won't show up in other_folder in the "Test Explorer" just from saving the file - but it should. Instead, manually triggering test discovery is required).

[alanwilter]

Well, I'm facing the opposite problem here. I DO want pytest. I had set it up and everything was working fine until this new "Discover tests..." were pushed in production.

I have the same repo in 3 different remote linuxes using the very same VScode (v1.83.1) and Python extension version (v2023.18.0). In one the "Discover tests..." WORKS!, in the other two it DOES NOT. The Testing Explore sidebar keeps always on and no meaningful messages are shown, no errors, nothing!:

No matter what I try.

How can I get the old "Discovery tests..." back please?

[eleanorjboyd]

@tboddyspargo, thanks for the detailed analysis! Yes, you are correct that [info] Discover tests for workspace name:... is printed even when neither testing framework is enabled (you can see this by removing all test configurations and clicking on the test tab and this still prints). This log is incorrect because it is not discovering, it will only do so if either pytest or unittest are enabled. I will make a change to the logs now to fix that message.

I have your provided repo open (thank you for including that) and see the following. This seems different to what you are describing since I do not have theworkspace_folder node in the Test Explorer sidebar. Also in the logs what I see is cwd: ~/testingFiles/from_users/workspace_folder_toml_/other_folder while your logs printed cwd: /tmp/workspace_folder. Which suggest our configurations may be different.

To confirm you are opening the folder as open workspace from file... with the multiroot file right?

I think maybe it is because we have different version of the extension, could you try on the newest release of the python extension? Preferably pre-release but stable might work too as I made some substantial changes from last stable release.

[tboddyspargo]

To confirm you are opening the folder as open workspace from file... with the multiroot file right?

Yes, that's correct.

I think maybe it is because we have different version of the extension, could you try on the newest release of the python extension? Preferably pre-release but stable might work too as I made some substantial changes from last stable release.

I'm on v2023.18.0, but I've also tried the pre-release version and I am seeing the same behavior as previously described in both cases.

@eleanorjboyd - Can you confirm that you are taking the extra step of opening the test_true.py file and saving it to trigger discovery in the other_folder workspace?

To summarize, the steps would be:

1. Open workspace_folder.zip using open workspace from file...
   • Test Explorer at this point will correctly display ONLY other_folder and the logs will corroborate that.
2. Open the test_true.py and save it.
3. Examine the Python output logs.
   • You should see something like this (notice the cwd and rootdir are workspace_folder, but I expected them to both be set as other_folder):

   2023-10-17 15:04:55.861 [info] > ~/.pyenv/versions/3.11.4/bin/python -m pytest -p vscode_pytest --collect-only
   2023-10-17 15:04:55.861 [info] cwd: /tmp/workspace_folder
   2023-10-17 15:04:56.064 [info] ============================= test session starts ==============================
   platform darwin -- Python 3.11.4, pytest-7.4.2, pluggy-1.3.0
   2023-10-17 15:04:56.064 [info] 
   rootdir: /private/tmp/workspace_folder
   
   2023-10-17 15:04:56.069 [info] collected 1 item
   
   2023-10-17 15:04:56.069 [info] 
   <Module other_folder/test_true.py>
     <Function test_dummy>
   
   2023-10-17 15:04:56.072 [info] Test server connected to a client.
   2023-10-17 15:04:56.073 [info] ResultResolver EOT received for discovery.
   2023-10-17 15:04:56.073 [info] 
   ========================== 1 test collected in 0.01s ===========================
   
   2023-10-17 15:04:56.095 [info] Disposing data receiver for /tmp/workspace_folder and deleting UUID; pytest discovery.
   

4. Examine the nodes in Test Explorer
   • You should see both other_folder (expected) and workspace_folder (unexpected)

[eleanorjboyd]

Aha! That was the missing piece, sorry that took a second @tboddyspargo. I have found the bug as we are not cycling through all workspaces for discovery like we do for testing. Apologies on how long that took and me skipping that step, these issues can get long and juggling a few of them gets confusing.

Anyways, Ill put in a fix now- it should fix most issues people are seeing with multiroot workspaces.

[tboddyspargo]

Aha! That was the missing piece, sorry that took a second @tboddyspargo.

Ill put in a fix now- it should fix most issues people are seeing with multiroot workspaces.

Fantastic! No worries - my journey to identifying the repro steps was probably hard to follow, too. I’m glad you were able to hone in on the problematic area so quickly!

[eleanorjboyd]

@tboddyspargo were you able to verify that the fix worked? No worries if not as I can also have a team member verify before we send this out to stable. Thanks

[tboddyspargo]

Thanks for circling back, @eleanorjboyd! When I saw that your PR had merged, I updated to the pre-release version and I can confirm that it did resolve the issue. I'll switch back to the stable release once this fix is incorporated!

[eleanorjboyd]

great thank you!