-
-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Missing Python packages due to cleared $PYTHONPATH
#22681
Comments
Hey @Flamefire, thanks for reporting. You said:
We haven't contemplated the possibility of people building a Python installation by adding different directories to
We remove
I don't think there is a simple way to prevent the problem I mentioned without dropping |
How exactly are they adding directories to it? IIRC I read in one of the related issues that it was about the current directory being used implicitly as a python search path. Maybe that could be detected and only that one removed from PYTHONPATH, potentially together with any empty values as IIRC |
Rather than |
Unfortunately that doesn't work for our use case: We have environment modules managed via LMod to allow users to choose their desired version of software in a (heavily) shared environment/filesystem. Those "modules" can be though of as (bash) scripts that are sourced and set/update and export environment variables. Hence the only thing a module providing a set of Python packages can do is append or prepend to In particular we cannot modify the global Python installation and put additional files in there, so the We are switching to a mechanism using IIRC the As the intention is that people "were very frustrated when it happened because it's not an easy problem to understand and find help about." maybe another solution is to detect a set I can also imaging another variable like |
@Flamefire, thank you for the deeper explanation. I think I understand better now what your situation is. How much control do you have over these module scripts? Could you do something like the following? |
@mrclary Thanks for the reply and suggestions
More or less full control. BUT:
I didn't know about that variable and it might come in handy.
So one solution I see is putting our And as confusing as it might be if users set up |
Yes, I see. That would require updating the EasyBuild and redeploy/reinstall across your clusters.
Yes, I see. Python also provides
I suppose that
Yep. Either the change is made with EasyBuild and redeploy/reinstall across your systems, or changes are made to Spyder and you update Spyder. How is Spyder installed in your situation? Does it have its own module or is it installed in the base Python location? Python 3.11 introduced the So, in conclusion, I agree that @Flamefire's suggestion to use an environment variable to switch |
Actually, I think this is possible, but still not convenient. |
Yes that is not possible because it is a shared filesystem, so that location needs to be at least specific to each user as they must not interfere with each other. But users can also have multiple independent tasks (from different shell sessions) running concurrently so that location must also be per-user-per-shell-session and still on the global filesystem which is impractical.
Correct. We have a shell variable that is prepended to by modules and that file reads it and adds the appropriate paths to
Yes it has it's own module. So updating that is much easier, especially as currently it cannot be used as a module, although we have a workaround: We remove the unset-code with a patch. But a general solution avoiding the need for patches going forward would be appreciated.
That seems to avoid the common issue with "wrong" files next to the file you want to execute. Interesting option indeed.
Just to be exact: Currently spider unsets |
|
True, I was just confused of what "ADD" would mean from the perspective of someone using |
I agree that that's the best we could do to allow this use case. |
I like this. 👍🏼 |
@Flamefire, If I submit a pull request, will you be able to help me test it on your system? I can, of course, test the state of |
Yes I can test that as time allows. So there might be a bit of delay but it isn't so urgent that it should be fine. |
Note: Could this be related/should supersede #21633 ? |
@Flamefire @smax1,
|
@mrclary Correct. And just to be clarify on the 2nd point: When started from a console on Linux PYTHONPATH from the environment is honored except paths matching a certain pattern And for both we want an option to disable any filtering |
Problem Description
We install Spyder alongside other globally installed Python packages that are in non-standard locations. To make Python find them we either add the locations to
$PYTHONPATH
or use a customsitecustomize.py
whose location must be in$PYTHONPATH
When starting Spyder it fails with errors such as
No module named 'zmq'
.I traced that to
spyder/spyder/app/start.py
Lines 13 to 18 in a857397
That removes the entries that are required such that Python finds the modules and our
sitecustomize
Removing that results in a similar problem as the IDE console shows
That arises due to removing
$PYTHONPATH
from the environment the kernel starts in:spyder/spyder/plugins/ipythonconsole/utils/kernelspec.py
Line 172 in a857397
I'm not fully sure what the intention was, but maybe that can be solved differently. Is that possible?
Versions
The text was updated successfully, but these errors were encountered: