Globs in extra paths

[itsdani]

I'm trying to setup a python monorepo build with bazel, and I'm struggling to make pylance recognize the library imports. I have managed to make it work by adding e.g. <bazel-myproject>/external/<my-pip-dependencies>/pypi__numpy to python.analysis.extraPaths, but adding a line for each dependency on every computer we use doesn't seem to be right.

My idea/request is that if we could add something like dependencies/*/ or dependencies/pypi__*/ to the extra paths, it would solve this problem entirely. The expected behavior would be to add every (matching) directory under dependencies/ to the extra paths.

[judej]

Thanks for the request @itsdani, we will review the request and look at what it would take to get this functionality

[jakebailey]

Copying my info here from microsoft/pyright#1684 (comment)

One consideration is that file watching logic for this becomes a little complicated because 1) we may need special handlers for parent directories of globbed items to know when the paths have to be regenerated, and 2) glob patterns may match files which we don't want in extraPaths (besides maybe zips if those ever get supported).

[dermidgen]

This is pretty critical for supporting monorepos with multiple python services.

Support for this feature would also resolve many of the open issues related to poetry with the use of local envs.

My teams currently address this challenge with a bash script that is run whenever a new python service or library is added.

#!/bin/bash
shopt -s dotglob
shopt -s nullglob

VENV_PATHS_ARR=(*/*/.venv/lib/*/site-packages/)
VENV_PATHS=$(printf ",\"%s\"" "${VENV_PATHS_ARR[@]}")
EXTRA_PATHS=${VENV_PATHS:1}

jq ".\"python.analysis.extraPaths\" = [${EXTRA_PATHS}]" .vscode/settings.json > .vscode/settings.tmp
mv .vscode/settings.tmp .vscode/settings.json 

[jakebailey]

Support for this feature would also resolve many of the open issues related to poetry with the use of local envs.

Can you clarify what you mean? I wouldn't expect anyone to use extraPaths to add paths that point to an interpreter.

[dermidgen]

We found extraPaths was the simplest way to support import resolution for small-medium monorepos. For larger monorepos or where granular ACLs are required, we found multi-root workspaces to be useful. However, even with multi-root workspaces, some roots may contain 10's to hundreds of services or libraries.

If there's a better approach than extraPaths - like actually switching envs - I would be very pleased to follow that. The downside to this approach is that imports will resolve regardless of which project your in.

With poetry and the following scaffolding:

./
...
./libs
./libs/lib-one
./libs/lib-one/poetry.toml
./libs/lib-one/pyproject.toml
./libs/lib-one/.venv
...
./stack
./stack/service-one
./stack/service-one/poetry.toml
./stack/service-one/pyproject.toml
./stack/service-one/.venv
...

Using poetry.toml

[virtualenvs]
in-project = true

[jakebailey]

You may be interested in https://github.com/microsoft/pyright/blob/main/docs/configuration.md#execution-environment-options, though I don't know to what extent it will have the behavior you're seeking. Execution environments are what @erictraut's team uses to manage their monorepo.

[dermidgen]

Execution environments are indeed helpful, in that it resolves the issue of resolving imports from outside the project path. This does require switching from .vscode/settings.json to pyproject.toml at the root of the repo - no biggie.

Unfortunately, since it doesn't allow setting venv for each execution environment 🤭, we still need to set extraPaths for each execution environment. Which, yes, is better than using .vscode/settings.json. However, since we need to set extraPaths we'll still need a helper script to do a glob drill down for "*/*/.venv/lib/*/site-packages/".

So I guess, if we can't get executionEnvironments to support an override for venv, we'd still greatly appreciate glob support for extraPaths.

[jakebailey]

The execution environment model is not suited to have separate interpreters, unfortunately. There used to be support, but it was broken as it was never intended to work. Multi-root runs separate analyzers, so they are able to have separate interpreters. Execution environments are within an analyzer, and can only have one.

We need to come up with a spec to better reconcile the idea of execution environments and multiroot into something that can capture how people are using. I think details on how your monorepo is laid out and what you're expecting would be helpful for that (though, this issue was about globbing, not that redesign 🙂)

[dermidgen]

@jakebailey I'd be happy to support that effort in any way; we can move the interpreters/monorepo discussion. Here's an example of what I'm currently testing wtih. https://github.com/dermidgen/python-monorepo

I yield the thread back to globbing 😁

[danmx]

any updates?

[judej]

Moving this issue to discussion as an enhancement request for comments and upvotes.