gh-57684: Add -P cmdline option and PYTHONSAFEPATH env var

[vstinner]

• Add -P command line option to not prepend an unsafe path
  to sys.path.
• Add sys.flags.safe_path flag.
• Add PyConfig.safe_path member.
• Programs/_bootstrap_python.c uses config.safe_path=0.
• Update subprocess._optim_args_from_interpreter_flags() to handle
  the -P command line option.
• Modules/getpath.py sets safe_path to 1 if a "._pth" file is
  present.
• Add PYTHONSAFEPATH environment variable.

https://bugs.python.org/issue13475

[vstinner]

https://docs.python.org/3/using/cmdline.html#cmdoption-c should be updated.

[CAM-Gerlach]

Thanks for this! This is a long-awaited and very useful change.

I was going to suggest fixes to some grammar errors and phrasing issues in the docs that were added, as well as an issue with the description in the man page sounding pretty cryptic for the average user, but I noticed this PR was still in a draft state. Is that just due to

https://docs.python.org/3/using/cmdline.html#cmdoption-c should be updated.

and it is otherwise ready for review? I'd be happy to help with that too, but it looks like a pretty small change, just updating it to reflect that an empty string rather than the working dir is added to sys.path[0].

[hroncok]

FTR I consider Don't add sys.path[0] a bit confusing, as there will always be some sys.path[0]. What about Don't prepend sys.path with anything?

[CAM-Gerlach]

FTR I consider Don't add sys.path[0] a bit confusing, as there will always be some sys.path[0]. What about Don't prepend sys.path with anything?

Yep, I noticed that as well above. I can't claim either of your expertise, but I'm not a beginner either and have actively been following the progress of and looking forward to this feature, and yet I had to carefully re-read the linked issue to be sure that it was actually doing what I thought it was. As both a developer and a documentarian, it can be easy for me to write documentation (or PEPs, etc) that makes perfect sense to me but is confusing to others without deep implicit knowledge of the code—I've seen both sides of that a lot lately, as well as the value of having someone else look over it.

[ofek]

Awesome! Is there an associated environment variable?

[CAM-Gerlach]

Awesome! Is there an associated environment variable?

It would appear there isn't, as of yet.

[vstinner]

I rebased my PR and marked it as ready for review ;-)

[ofek]

Can we add an associated environment variable?

[vstinner]

Since different people asked for an environment variable, I changed my mind and I added PYTHONDONTADDPATH0 environment variable. Example:

$ ./python -c 'import sys, pprint; pprint.pprint(sys.path)'
['',
 '/usr/local/lib/python311.zip',
 '/home/vstinner/python/main/Lib',
 '/home/vstinner/python/main/build/lib.linux-x86_64-3.11-pydebug',
 '/home/vstinner/.local/lib/python3.11/site-packages']

$ PYTHONDONTADDPATH0=1 ./python -c 'import sys, pprint; pprint.pprint(sys.path)'
['/usr/local/lib/python311.zip',
 '/home/vstinner/python/main/Lib',
 '/home/vstinner/python/main/build/lib.linux-x86_64-3.11-pydebug',
 '/home/vstinner/.local/lib/python3.11/site-packages']

[eryksun]

Suggested change

	Don't add :data:`sys.path[0] <sys.path>`: don't prepend the current working
	directory (``python -m module``), an empty string (``python -c code``) or
	the first command line argument (``python script.py``) to :data:`sys.path`.
	Do not prepend the initial working directory (``python -m module``), the
	current working directory (``python -c code``), or the directory of the
	executed script (``python script.py``) to :data:`sys.path`.

I think saying "prepend" suffices instead of talking about adding sys.path[0].

With the -m option, sys.path[0] is the full path of the initial working directory. With the -c option and the REPL, the current working directory is prepended to sys.path. It's an implementation detail that it's represented in this case as an empty string. It's not exactly an obscure implementation detail, since most people know that normpath('') -> '.', but I think it's better to be explicit about the behavior.

[eryksun]

You're emphasizing sys.path[0] again. There's always a sys.path[0]. How about PYTHONUSESAFEPATH?

[merwok]

How does the doc call the value that is prepended? Main script base directory? Could be PYTHONDONTADDMAINDIR or PYTHONNOMAINDIR.

(I wondered about including IMPORT or PATH but then you get something clunky or you add prepositions and get too long)

[vstinner]

Proposed names so far (on python-dev and this PR), I add spaces for readability:

• PYTHON DONT ADD PATH0
• PYTHON DONT ADD PWD
• PYTHON DONT ADD MAIN DIR
• PYTHON DONT ADD IMPLICT DIRS
• PYTHON DONT ADD SCRIPT DIR
• PYTHON USE SAFE PATH

[vstinner]

Proposed long option names (in the issue):

• "main path": --mainpath and --nomainpath
• "base path": --basepath <dir> and --no-basepath: https://bugs.python.org/issue33053#msg314192
• "path 0": --nopath0 with short option -p

[vstinner]

Perl 5.26 no longer includes the current working directly by default in @INC to fix CVE-2016-1238 security vulnerability: https://metacpan.org/pod/perl5260delta#Removal-of-the-current-directory-(%22.%22)-from-@INC

PERL_USE_UNSAFE_INC env var can be used to opt-in for "unsafe" @INC (add current working directory).

Perl also has a "taint mode" enabled by the Perl -T command line option.

[vstinner]

I updated my PR: it now uses the PYTHONSAFEPATH environment variable name.

• Replace bpo-13475 with gh-57684
• Rename add_path0 to safe_path (opposite meaning)
• Rename sys.flags.dont_add_path0 to sys.flags.safe_path
• Rename PYTHONDONTADDPATH0 env var to PYTHONSAFEPATH (shorter, easier to write)
• PR rebased, I fixed conflicts.

[hroncok]

I love PYTHONSAFEPATH!

[JelleZijlstra]

I'm not a fan of the "safe path" wording. Things can be "safe" or "unsafe" in many ways and the word doesn't give a clear indication of what the precise behavior is.

Would something like "skip_cwd_in_path" work?

[merwok]

It wouldn’t, because as detailed before it’s not always the working dir that’s added.

[zware]

I'm not a fan of the "safe path" wording. Things can be "safe" or "unsafe" in many ways and the word doesn't give a clear indication of what the precise behavior is.

Same reservations here. What about changing the sense of the name, to PYTHONUNSAFEPATH/sys.flags.unsafe_path with a default of True? Not a strong enough feeling to be blocking, though :)

[vstinner]

@gpshead: Would you mind to review the updated PR?

[vstinner]

@hroncok: Thanks, I addressed your review on the doc.

[zware]

provide an opt-in way "to be safe".

That's the bit that makes me squirm, though :). It makes things less potentially unsafe, but I can well imagine someone coming along in 5 years with some new contrivance that shows that no, in fact, the path is not completely safe with PYTHONSAFEPATH set.

Maybe make it PYTHONSAFERPATH? That wording is harder to argue against, but also hard to tell apart from PYTHONSAFEPATH.

I fully recognize that this is just bikeshedding, though. Don't hold this up on quibbles about future maybes :)

[vstinner]

Maybe we should consider adding -p option which is the opposite of -P: add the unsafe path.

For example, PYTHONSAFEPATH=1 python -p script.py [...] would ensure that Python still adds the current directory to sys.path to run script.py. The problem is that when you run a script, you don't know in advance in PYTHONSAFEPATH=1 env var is set or not: it's inherited in the parent process environment or can be set system-wide.

Otherwise, script.py fails on import foo to load foo.py which is in the same directory than script.py.

[vstinner]

Maybe we should consider adding -p option which is the opposite of -P: add the unsafe path.

For example, PYTHONSAFEPATH=1 python Tools/freeze/freeze.py fails on import checkextensions with ImportError: it is supposed to load Tools/freeze/checkextensions.py. The freeze.py script is incompatible with "safe path".

[merwok]

These tools are exceptions; they could add their parent directory to sys.path explicitly.

[gpshead]

Maybe we should consider adding -p option which is the opposite of -P: add the unsafe path.

I think that should be left for the future. Lets see if anyone actually wants it first. If we do go the route of aiming for a default flip in the future it might make sense.

[ofek]

I'm still thinking about changing the default to be safe by default, but "later" and it might require a PEP.

I'd be very much in favor of such a change 🙂

[vstinner]

That's the bit that makes me squirm, though :). It makes things less potentially unsafe, but I can well imagine someone coming along in 5 years with some new contrivance that shows that no, in fact, the path is not completely safe with PYTHONSAFEPATH set.

PYTHONSAFEPATH=1 is not safe at all :-) It does exactly one thing: not preprend a path to sys.path[0] which is "potentially" unsafe. It's just that it's really hard to select a short environment name which either summarizes well what the variable does, or explains in length what it does. For example, the PYTHONDONTADDIMPLICTDIRS name was proposed, but ... it's quite long to type :-(

[vstinner]

@gpshead:

I think that should be left for the future. Lets see if anyone actually wants it first. If we do go the route of aiming for a default flip in the future it might make sense.

Flipping the default can now be done by setting PYTHONSAFEPATH=1 system wide (or in the current shell).

The problem is that right now, if PYTHONSAFEPATH=1 is set, it's no longer possible to run programs which rely on the Python 3.10 behavior to load modules in the current directory (as Python freeze.py tool), without having to modify the source code of these programs.

IMO command line options must have the priority over environment variables. Python does not provide command line options to disable the effect of all Python environment variables. For example, PYTHONUNBUFFERED=1 cannot be "undone" by a command line option. But the effect of this variable is limited: I'm not aware of any program broken by it.

The Python UTF-8 Mode is way more likely to cause compatibility issues and so -X utf8=0 can disable it on the command line: ignore PYTHONUTF8=1 env var.

We don't have to add -p, but it makes PYTHONSAFEPATH=1 harder to use. The current workaround is to use -E to ignore environment variables, or -I (isolated mode) but this option has more effects (like not adding user site directories).

[vstinner]

I merged my PR, thanks @gpshead for the review.

I'm not fully comfortable of merging this PR in a rush (before the Python 3.11 feature freeze), but IMO the scope of the feature is limited and well defined. It's different than other options discussed previously, especially the idea of changing the default behavior to "safe" (not prepend a path to sys.path).

The idea of putting the feature is beta1 is to give users the ability to play with it before Python 3.11 final release, to get feedback, adjust issues, and maybe consider reverting the change if it causes too many issues.

[vstinner]

@ofek:

I'd be very much in favor of such a change slightly_smiling_face

As written previously, you can now simply set PYTHONSAFEPATH=1 env var system wide, and Python 3.11 sys.path will be "safer" by default ;-) On Linux, just put it in /etc/environment.

[vstinner]

Follow-up change: 329afe7 "Fix tests failing with the PYTHONSAFEPATH=1 env var".

[vstinner]

I created #92361 to add a new -p option which can be used to opt-in for Python 3.10 sys.path behavior when PYTHONSAFEPATH=1 is set globally. It can also be used with the -I isolated mode to prepend the unsafe path to sys.path.