Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add a mechanism to disable the GIL #116167

Closed
Tracked by #108219
swtaarrs opened this issue Feb 29, 2024 · 8 comments
Closed
Tracked by #108219

Add a mechanism to disable the GIL #116167

swtaarrs opened this issue Feb 29, 2024 · 8 comments
Assignees
@swtaarrs
Labels
interpreter-core (Objects, Python, Grammar, and Parser dirs) topic-free-threading type-feature A feature request or enhancement

Comments

@swtaarrs
Copy link
Member

swtaarrs commented Feb 29, 2024 •
edited
Loading

Feature or enhancement

Proposal:

PYTHON_GIL=0 python ... or python -Xgil=0 ... should disable the GIL at runtime, in Py_GIL_DISABLED builds. This will be similar in spirit to colesbury/nogil-3.12@f546dbf16a.

Has this already been discussed elsewhere?

I have already discussed this feature proposal on Discourse

Links to previous discussion of this feature:

https://peps.python.org/pep-0703/

Linked PRs
@swtaarrs swtaarrs added type-feature A feature request or enhancement topic-free-threading labels Feb 29, 2024
@swtaarrs swtaarrs self-assigned this Feb 29, 2024
@swtaarrs swtaarrs mentioned this issue Feb 29, 2024
Open
@itamaro
Copy link
Contributor

itamaro commented Feb 29, 2024

shouldn't the GIL be disabled by default in free-threading builds? (and then have -x withgil or PYTHON_GIL=1 to enable it)

@swtaarrs
Copy link
Member Author

@colesbury probably has more thoughts, but I believe we're expecting disabling the GIL to be pretty buggy initially. So even if we're targeting disabling the GIL by default in 3.13, we'll probably want the GIL to stay on by default for a while first.

@colesbury
Copy link
Contributor

colesbury commented Feb 29, 2024 •
edited
Loading

I think we want to work towards the behavior described in PEP 703 with the Py_mod_gil slot and PTYHONGIL 1 environment variable: https://peps.python.org/pep-0703/#py-mod-gil-slot.

As a starting point:

  1. We should keep the GIL enabled by default in free-threaded builds (as it is currently is) for now.
  2. PYTHON_GIL=0 should disable the GIL

We can add the Py_mod_gil slot and eventually disable the GIL by default later on, but hopefully still before the 3.13 release.

EDIT: There seemed to be a strong preference for UNDERSCORES_IN_NAMES in https://discuss.python.org/t/change-environment-variable-style/35180

Footnotes

  1. Or PYTHON_GIL -- I think people have expressed a preference for _ separated names in new environment variables ↩

@swtaarrs swtaarrs added the interpreter-core (Objects, Python, Grammar, and Parser dirs) label Mar 4, 2024
This was referenced Mar 4, 2024
Closed
Merged
colesbury pushed a commit that referenced this issue Mar 11, 2024
@swtaarrs
gh-116167: Allow disabling the GIL with PYTHON_GIL=0 or -X gil=0 (#…
2731913 
…116338)

In free-threaded builds, running with `PYTHON_GIL=0` will now disable the
GIL. Follow-up issues track work to re-enable the GIL when loading an
incompatible extension, and to disable the GIL by default.

In order to support re-enabling the GIL at runtime, all GIL-related data
structures are initialized as usual, and disabling the GIL simply sets a flag
that causes `take_gil()` and `drop_gil()` to return early.
@jakirkham
Copy link

The PEP references PYTHONGIL whereas PR ( #116338 ) adds PYTHON_GIL

Is there a difference between these two? Or is there a plan to consolidate to with or without _?

Sorry I know this is a minor question, but want to help avoid potential confusion down the road

@itamaro
Copy link
Contributor

itamaro commented Mar 12, 2024

The PEP references PYTHONGIL whereas PR ( #116338 ) adds PYTHON_GIL

Is there a difference between these two? Or is there a plan to consolidate to with or without _?

there's no difference - underscored env vars is the current preference among core devs (discussion & poll)

the implementation & documentation are the source of truth, and may differ from the PEP in some ways.

@gvanrossum
Copy link
Member

IIUC the python binaries distributed for Mac and Windows on python.org for 3.13 will not be built with --disable-gil, right? And IIUC we intend for Linux distros also to have binaries named python3.13, python3 and possibly python that are built without that flag, giving the option to also distribute binaries built with --disable-gil under different names. So this env var only applies to the latter, right? (For 3.14 the situation may be different.)

@colesbury
Copy link
Contributor

The environment variable PYTHON_GIL and -X gil=0 option only apply to the free-threaded binaries. Setting them with the default build cause an error on startup 1.

The Windows installer can optionally also install the free-threaded executable python3.13t.exe along with the default (with GIL) 3.13 executable. The Windows embeddable package only contains the default build (no free-threaded binaries).

Example Windows installation 
    Directory: C:\Users\coles\AppData\Local\Programs\Python\Python313


Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
d-----         3/13/2024   2:30 PM                DLLs
d-----         3/13/2024   2:30 PM                Doc
d-----         3/13/2024   2:30 PM                include
d-----         3/13/2024   2:30 PM                Lib
d-----         3/13/2024   2:30 PM                libs
d-----         3/13/2024   2:30 PM                Scripts
d-----         3/13/2024   2:30 PM                tcl
-a----         3/12/2024  10:10 PM          33861 LICENSE.txt
-a----         3/12/2024  10:12 PM        1841840 NEWS.txt
-a----         3/12/2024  10:10 PM         103192 python.exe
-a----         3/12/2024  10:11 PM         103192 python3.13t.exe
-a----         3/12/2024  10:10 PM          69912 python3.dll
-a----         3/12/2024  10:10 PM        6829336 python313.dll
-a----         3/12/2024  10:11 PM        6903064 python313t.dll
-a----         3/12/2024  10:11 PM          70936 python3t.dll
-a----         3/12/2024  10:10 PM         101656 pythonw.exe
-a----         3/12/2024  10:11 PM         101656 pythonw3.13t.exe
-a----         3/12/2024  10:10 PM         119192 vcruntime140.dll
-a----         3/12/2024  10:10 PM          49528 vcruntime140_1.dll

The macOS installer does not currently support installing the free-threaded build. I'm not sure what the plan is for 3.13.

Your summary of the Linux situation matches my understanding.

Footnotes

  1. We may want to improve the error message. It's currently:
    Fatal Python error: config_read_gil: PYTHON_GIL / -X gil are not supported by this build
    Python runtime state: preinitialized ↩

@swtaarrs
Copy link
Member Author

Once gh-116749 is complete, we can start running tests in CI with the GIL disabled using one of these flags (or by doing gh-116329).

This is finished unless someone wants to keep it open to track improving the error message (from Sam's last comment).

@serhiy-storchaka serhiy-storchaka mentioned this issue Mar 19, 2024
Closed
adorilson pushed a commit to adorilson/cpython that referenced this issue Mar 25, 2024
@swtaarrs @adorilson
pythongh-116167: Allow disabling the GIL with PYTHON_GIL=0 or `-X g…
819eb99 
…il=0` (python#116338)

In free-threaded builds, running with `PYTHON_GIL=0` will now disable the
GIL. Follow-up issues track work to re-enable the GIL when loading an
incompatible extension, and to disable the GIL by default.

In order to support re-enabling the GIL at runtime, all GIL-related data
structures are initialized as usual, and disabling the GIL simply sets a flag
that causes `take_gil()` and `drop_gil()` to return early.
diegorusso pushed a commit to diegorusso/cpython that referenced this issue Apr 17, 2024
@swtaarrs @diegorusso
pythongh-116167: Allow disabling the GIL with PYTHON_GIL=0 or `-X g…
ee3410d 
…il=0` (python#116338)

In free-threaded builds, running with `PYTHON_GIL=0` will now disable the
GIL. Follow-up issues track work to re-enable the GIL when loading an
incompatible extension, and to disable the GIL by default.

In order to support re-enabling the GIL at runtime, all GIL-related data
structures are initialized as usual, and disabling the GIL simply sets a flag
that causes `take_gil()` and `drop_gil()` to return early.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@swtaarrs swtaarrs

Labels
interpreter-core (Objects, Python, Grammar, and Parser dirs) topic-free-threading type-feature A feature request or enhancement
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@swtaarrs @itamaro @colesbury @gvanrossum @jakirkham