Handle python.pythonPath deprecation in the Microsoft Python extension

[sambhare]

Is your feature request related to a problem? Please describe.
The python.pythonPath setting is being deprecated in the Microsoft Python extension. Currently Pyright seems (?) to use this setting to select the correct virtual environment. Removing python.pythonPath from my .vscode/settings.json file breaks Pyright's selection of virtual environment.

Describe the solution you'd like
Update the Pyright VS Code extension to handle this change in the Microsoft Python extension.

[jakebailey]

This should be similar to the fix applied to the Code Runner extension: https://github.com/formulahendry/vscode-code-runner/blob/2bed9aeeabc1118a5f3d75e47bdbcfaf412765ed/src/utility.ts#L6

[jakebailey]

Well, that, and the LSP middleware change to ensure the server always sees the setting anyway: https://github.com/microsoft/vscode-python/pull/11084/files#diff-ed95fdd33556d8f532e3324d9fb399c9R87

[erictraut]

This will be implemented in the next release of Pyright.

[erictraut]

This is now implemented in version 1.1.45, which I just published.

[sambhare]

I'm trying it but I get this message after I remove python.pythonPath from .vscode/settings.json

[Info - 9:41:19 a.m.] Pyright language server 1.1.45 starting
[Info - 9:41:19 a.m.] Server root directory: c:\Users\Rajas.vscode\extensions\ms-pyright.pyright-1.1.45
[Info - 9:41:19 a.m.] Background analysis root directory: c:\Users\Rajas.vscode\extensions\ms-pyright.pyright-1.1.45
[Info - 9:41:19 a.m.] Background analysis started
Python was not found but can be installed from the Microsoft Store: https://go.microsoft.com/fwlink?linkID=2082640Python was not found but can be installed from the Microsoft Store: https://go.microsoft.com/fwlink?linkID=2082640

Am I missing a setting somewhere?

Version: 1.46.1 (system setup)
Commit: cd9ea6488829f560dc949a8b2fb789f3cdc05f5d
Date: 2020-06-17T21:13:20.174Z
Electron: 7.3.1
Chrome: 78.0.3904.130
Node.js: 12.8.1
V8: 7.8.279.23-electron.0
OS: Windows_NT x64 10.0.19041

[erictraut]

Hmm, that error message isn't being printed by Pyright. It seems to indicate that you don't have the Python extension installed (or enabled in your workspace). The new pythonPath mechanism requires it to be installed. Can you double check that you have it installed?

When I was implementing and testing this feature, it wasn't clear to me how to completely disable the "python.pythonPath" setting. While this setting is being deprecated, the Python extension's environment selection feature appears to write the path to the selected Python environment to both the old "python.pythonPath" setting as well as the new extension-specific config store. I presume that at some point it will no longer update the old setting.

[sambhare]

I do have both the Microsoft Python extension and Pyright installed.

I suspect it has to do with extension startup order. I think Pyright starts up before the Microsoft Python extension has started up and actually "loaded" my virtual environment.

I did the following to test this theory:

1. Open a Python file/folder with a venv setup
2. Disable Pyright
3. Reload VS Code
4. Wait for the Microsoft Python extension to finish loading and to finish loading the venv
5. Enable Pyright

PyRight seems to work properly now and there are no error messages.

[erictraut]

OK, thanks for those clues. I do have code in place that waits for the Python extension to start, but perhaps it's not working as intended. I'll look into it.

[jakebailey]

I think this is what happens when you just run bare "python" on Windows; "python" is this magic binary which points the user to install something on the Windows Store, then is replaced once that's done.

I've noticed this in testing in CI where I can see it printing out "Python 2.7.15" or similar in tests because "python" is being executed; I'd be curious if it's still being over-run when things are unset (where the most recent fixes were intended to only run Python if pythonPath was set), or some bug where the wrong executable is executed.

[erictraut]

I think Jake is correct that the message you're seeing is because no pythonPath was found, so Pyright is attempting to execute "python" as a fallback mechanism. With that mystery solved, we now need to understand why pythonPath isn't being read from the Python extension. I have code in place that should wait until the Python extension is activated, and it appears to work on my machine. Perhaps the problem is timing-related or is specific to Windows. I've added more logging to the Output window to help us diagnose it further.

[sambhare]

Do you wait until the Python extension is activated or until the Python extension has activated and activated the virtual environment? To ask another way, does Pyright get notified if someone clicks on the status bar and changes the Python virtual environment in use?

[erictraut]

I wait until the extension is activated. The Python extension doesn't currently expose any mechanism for notifying when someone activates or changes a virtual environment. We're looking at whether the Python extension could add an event of some sort.

[sambhare]

I think that might be the issue here. And it might affect more than just Pyright. As a workaround, would be it possible to add a command to Pyright e.g. "Reload Pyright pythonPath" or something like that that I could access from the command palette?

[erictraut]

That could explain what's happening. I'm not able to repro it on my machine, but it could be timing-related. I've added a bunch of additional logging that will be in the next version of Pyright to help us diagnose it further.

Pyright already has a command for restarting the language server ("Pyright: Restart Server"). This is effectively what happens when a new pythonPath is received.

[erictraut]

I just published Pyright 1.1.46, which contains additional logging to the output pane. @sambhare, could you please try to repro the problem with this new version and let me know what output you see? Thanks!

[sambhare]

On a fresh start of VS Code and opening a folder with a venv set, I see this

[Info  - 11:12:18 a.m.] Pyright language server 1.1.46 starting
[Info  - 11:12:18 a.m.] Server root directory: c:\Users\Rajas\.vscode\extensions\ms-pyright.pyright-1.1.46
[Info  - 11:12:18 a.m.] Background analysis root directory: c:\Users\Rajas\.vscode\extensions\ms-pyright.pyright-1.1.46
[Info  - 11:12:18 a.m.] Background analysis started
Received pythonPath from Python extension: python
Python was not found but can be installed from the Microsoft Store: https://go.microsoft.com/fwlink?linkID=2082640Python was not found but can be installed from the Microsoft Store: https://go.microsoft.com/fwlink?linkID=2082640

(I don't have python.exe on my Windows PATH by design as I always work in venvs)

If I do Pyright: Restart Server, nothing more is printed

If I disable Pyright, reload VS Code, wait for Microsoft Python to show my venv on the status bar and then enable Pyright again I see this and it works:

[Info  - 11:15:56 a.m.] Pyright language server 1.1.46 starting
[Info  - 11:15:56 a.m.] Server root directory: c:\Users\Rajas\.vscode\extensions\ms-pyright.pyright-1.1.46
Received pythonPath from Python extension: C:\Users\Rajas\Code\venv\py38api\Scripts\python.exe

[jakebailey]

Could be the auto interpreter picking; I'd be curious to see the Python extension's logs to see if it's picking the default and then changing it after.

[sambhare]

Python extension logs on fresh start

User belongs to experiment group 'AlwaysDisplayTestExplorer - control'
User belongs to experiment group 'ShowPlayIcon - start'
User belongs to experiment group 'DebugAdapterFactory - experiment'
User belongs to experiment group 'PtvsdWheels37 - experiment'
User belongs to experiment group 'UseTerminalToGetActivatedEnvVars - control'
User belongs to experiment group 'LocalZMQKernel - control'
User belongs to experiment group 'CollectLSRequestTiming - control'
User belongs to experiment group 'CollectNodeLSRequestTiming - experiment'
User belongs to experiment group 'EnableIPyWidgets - experiment'
User belongs to experiment group 'DeprecatePythonPath - experiment'
User belongs to experiment group 'RunByLine - control'
User belongs to experiment group 'CustomEditorSupport - control'
> conda --version
> pyenv root
> python3.7 c:\Users\Rajas\.vscode\extensions\ms-python.python-2020.6.90262\pythonFiles\pyvsc-run-isolated.py -c "import sys;print(sys.executable)"
> python3.6 c:\Users\Rajas\.vscode\extensions\ms-python.python-2020.6.90262\pythonFiles\pyvsc-run-isolated.py -c "import sys;print(sys.executable)"
> python3 c:\Users\Rajas\.vscode\extensions\ms-python.python-2020.6.90262\pythonFiles\pyvsc-run-isolated.py -c "import sys;print(sys.executable)"
> python2 c:\Users\Rajas\.vscode\extensions\ms-python.python-2020.6.90262\pythonFiles\pyvsc-run-isolated.py -c "import sys;print(sys.executable)"
> python c:\Users\Rajas\.vscode\extensions\ms-python.python-2020.6.90262\pythonFiles\pyvsc-run-isolated.py -c "import sys;print(sys.executable)"
> py -3.7 c:\Users\Rajas\.vscode\extensions\ms-python.python-2020.6.90262\pythonFiles\pyvsc-run-isolated.py -c "import sys;print(sys.executable)"
> py -3.6 c:\Users\Rajas\.vscode\extensions\ms-python.python-2020.6.90262\pythonFiles\pyvsc-run-isolated.py -c "import sys;print(sys.executable)"
> py -3 c:\Users\Rajas\.vscode\extensions\ms-python.python-2020.6.90262\pythonFiles\pyvsc-run-isolated.py -c "import sys;print(sys.executable)"
> py -2 c:\Users\Rajas\.vscode\extensions\ms-python.python-2020.6.90262\pythonFiles\pyvsc-run-isolated.py -c "import sys;print(sys.executable)"
> python c:\Users\Rajas\.vscode\extensions\ms-python.python-2020.6.90262\pythonFiles\pyvsc-run-isolated.py -c "import sys;print(sys.executable)"
> conda info --json
> python c:\Users\Rajas\.vscode\extensions\ms-python.python-2020.6.90262\pythonFiles\pyvsc-run-isolated.py -c "import sys;print(sys.executable)"
> python c:\Users\Rajas\.vscode\extensions\ms-python.python-2020.6.90262\pythonFiles\pyvsc-run-isolated.py -c "import sys;print(sys.executable)"
Python interpreter path: ~\Code\venv\py38api\Scripts\python.exe
Starting Microsoft Python language server.
> ~\Code\venv\py38api\Scripts\python.exe c:\Users\Rajas\.vscode\extensions\ms-python.python-2020.6.90262\pythonFiles\pyvsc-run-isolated.py flake8 --ignore=E501 --max-complexity=10 --extend-ignore=ANN101,ANN102 --format=%(row)d,%(col)d,%(code).1s,%(code)s:%(text)s c:\Users\Rajas\Code\Nuralogix\dfx-api-client-python-async\apiexample.py
cwd: c:\Users\Rajas\Code\Nuralogix\dfx-api-client-python-async
> ~\Code\venv\py38api\Scripts\python.exe c:\Users\Rajas\.vscode\extensions\ms-python.python-2020.6.90262\pythonFiles\pyvsc-run-isolated.py flake8 --ignore=E501 --max-complexity=10 --extend-ignore=ANN101,ANN102 --format=%(row)d,%(col)d,%(code).1s,%(code)s:%(text)s c:\Users\Rajas\Code\Nuralogix\dfx-api-client-python-async\apiexample.py
cwd: c:\Users\Rajas\Code\Nuralogix\dfx-api-client-python-async
> conda --version
##########Linting Output - flake8##########

Snipped the actual flake output.Apparently I'm in a few telemetry experiments.

Python language server output

[Info  - 2:08:47 p.m.] Analysis cache path: C:\Users\Rajas\AppData\Local\Microsoft\Python Language Server
[Info  - 2:08:47 p.m.] Microsoft Python Language Server version 0.5.45.0
[Info  - 2:08:47 p.m.] Workspace root: c:\Users\Rajas\Code\Nuralogix\dfx-api-client-python-async
[Info  - 2:08:48 p.m.] GetCurrentSearchPaths C:\Users\Rajas\Code\venv\py38api\Scripts\python.exe 
[Info  - 2:08:48 p.m.] Interpreter search paths:
[Info  - 2:08:48 p.m.]     c:\program files\python38\dlls
[Info  - 2:08:48 p.m.]     c:\program files\python38\lib
[Info  - 2:08:48 p.m.]     c:\program files\python38
[Info  - 2:08:48 p.m.]     c:\users\rajas\code\venv\py38api
[Info  - 2:08:48 p.m.]     c:\users\rajas\code\venv\py38api\lib\site-packages
[Info  - 2:08:48 p.m.]     c:\users\rajas\code\nuralogix\dfx_apiv2_protos_python
[Info  - 2:08:48 p.m.]     c:\users\rajas\code\nuralogix\dfxpydemo
[Info  - 2:08:48 p.m.] User search paths:
[Info  - 2:08:49 p.m.] Initializing for C:\Users\Rajas\Code\venv\py38api\Scripts\python.exe
[Info  - 2:08:49 p.m.] Analysis caching mode: None.

I think the Python extension seems to be searching for Python on my machine until it finally gets to the one I've set in my virtualenv.

[jakebailey]

Odd, doesn't say anything about just "python". Could be that it's the "default" setting being picked up (the extension has that notion internally, and it might be "python"). Unfortunately the logs don't state the reason an interpreter was chosen.

You should probably set your python.languageServer setting to "None" so that MPLS doesn't run; it'd be redundant.

[erictraut]

OK, looks like it's safe to say that the Python extension is initially returning "python" until it finds the selected venv. When it does, it has no way of notifying other extensions. Until a notification mechanism is added, I don't see a good way to fix this. I suppose we could add a periodic polling mechanism, but that has a bunch of downsides. We'll just leave this bug open for now and wait for a notification mechanism to be implemented in the Python extension.

[erictraut]

@karthiknadig & @jakebailey, I implemented support for the new onDidChangeExecutionDetails notification callback in the Python extension. Works great! If you see anything wrong with my PR or have suggestions for a better approach, let me know. c89b6a5

[jakebailey]

The commit appears to do: languageClient.sendRequest('workspace/executeCommand', {}); when I believe it meant to send didChangeConfiguration.

[jakebailey]

In the core extension, I did this in a type safe manner without the string: https://github.com/microsoft/vscode-python/pull/12366/files#diff-53a0b543317dd8025be737a600e0fbd5R113

[erictraut]

Good catch. The comment was right, the code was wrong. I like your solution better, so I'm going to copy that. I'm surprised my previous code worked as well as it appeared to.

[erictraut]

This is now implemented in Pyright 1.1.55, which I just published.

[timothybrooks]

With version 1.1.55, the proper conda env set to the interpreter, and no value specified for pythonPath in settings.json, I am getting the following Pyright errors for third party modules:
Import "[name]" could not be resolved Pyright (reportMissingImports)
Is Pyright using a different python than the interpreter? When I specify pythonPath in settings.json to that of the proper conda env, Pyright correctly resolves these imports (until the pythonPath setting is removed on the next startup due to its deprecation).

[erictraut]

@timothybrooks, what version of the Python extension are you running? Is it the latest (v2020.7.94776) or newer?

[timothybrooks]

Yes, I am using ms-python.python v2020.7.94776 and ms-pyright v1.1.55

[erictraut]

@timothybrooks, you asked "Is Pyright using a different python than the interpreter?" It is meant to be using the same interpreter that is selected in the Python extension using its interpreter selector. Is that what you mean?

As you're aware, the Python extension previously stored the path to the selected interpreter in the python.pythonPath setting, but this is being deprecated. The latest version of the Python extension provides an interface that allows other extensions (like Pyright) to query the current selection and be notified when it changes. The latest version of Pyright implements this, so it should be picking up the interpreter path through this interface. It seems to be working for me, so if it's not working for you, let's figure out why.

If you open the Output panel and choose "Pyright", you should see a log output that looks something like this:
Received pythonPath from Python extension: /venvs/p38/bin/python. Does that appear? If you then select a different interpreter, you should see a similar log output with the new path. Does that appear?

I'll mention that one workaround is to switch from the Pyright extension to Pylance. Pylance supports all of the functionality of Pyright but is more tightly integrated with the Python extension, so it doesn't need to use this interface to query the selected python interpreter.

[timothybrooks]

Thanks for clarifying what's going on. Here is the Pyright output:

[Info  - 10:26:55 AM] Pyright language server 1.1.55 starting
[Info  - 10:26:55 AM] Server root directory: /home/tim/.vscode/extensions/ms-pyright.pyright-1.1.55
Waiting for Python extension to load
[Info  - 10:26:55 AM] Background analysis root directory: /home/tim/.vscode/extensions/ms-pyright.pyright-1.1.55
[Info  - 10:26:55 AM] Background analysis started
Python extension loaded
Received pythonPath from Python extension: python
[Error - 10:26:56 AM] stubPath /home/tim/code/mode_encoder/typings is not a valid directory

I see no additional output when selecting a new interpreter. (And regarding the stubPath error, I do not specify any stubPath in my settings).

Thanks for the help!

[erictraut]

The behavior you're reporting matches the behavior of older versions of the Python extension, prior to v2020.7.94776. That was the first version that introduced the new notification mechanism.

A newer version of the Python extension (v2020.7.96456) was released today. Please give that a try. If you're still seeing this behavior, please look in the "Python" output to see if there are any clues. It should log something like "Python interpreter path: xxx" when you select a new interpreter from the selector.