Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Python 3.4+: use os.set_inheritable + os.set_blocking in place of unreadable fcntl calls #3210

Open
wants to merge 17 commits into
base: master
Choose a base branch
from

Conversation

pajod
Copy link
Contributor

@pajod pajod commented May 12, 2024

We should cleanup those rather unreadable low-level fcntl calls. The stdlib provides single-call functions for these now.

(Replacement is not 100% equivalent, as they might call ioctl(.., FIO.., NULL) instead, but I see no reason we should care.)


Suggested order:

@pajod pajod marked this pull request as ready for review May 12, 2024 22:34
@benoitc benoitc self-assigned this Jun 14, 2024
@pajod pajod force-pushed the patch-fcntl-FD_CLOEXEC-ioctl-FIOCLEX branch from 3ab0385 to bc4fc46 Compare July 4, 2024 13:23
@pajod pajod marked this pull request as draft August 6, 2024 14:46
@benoitc
Copy link
Owner

benoitc commented Aug 8, 2024

Not disagreeing about this change but this must be validated on BSD. I think we need to add FreeBSD to our test platform to ensure it works there. (an dmaybe other OpenBSD).

@pajod pajod force-pushed the patch-fcntl-FD_CLOEXEC-ioctl-FIOCLEX branch 8 times, most recently from b5a25e9 to 4831ef3 Compare August 23, 2024 20:39
@pajod
Copy link
Contributor Author

pajod commented Aug 23, 2024

@benoitc Happy with pulling Neil Pang's vm images?

Those work with very little setup required for FreeBSD, OpenBSD and illumos (no prebuilt gevent though). Though the Github file still needs review by someone more proficient with that:

  • I did not know how to express "whatever python3 version you usually do" on those images
  • I did not know if/how mixing GitHub CI secrets with running arbitrary untrusted code in another CI workflow can be done safely.
  • The actual test I meant to run did not run on those systems yet (thats what the s in tests/test_nginx.py s indicates) - my current nginx test does not produce very nice messages when failing
  • afaict, OmniOS does not ship gevent, so setup including compilation is rather slow / does not work because libev believes it is on linux and tries to prepare for the existing inotify.h with two-argument statfs() needs some autoconf override in the CI workflow

@pajod pajod force-pushed the patch-fcntl-FD_CLOEXEC-ioctl-FIOCLEX branch 2 times, most recently from 5632dc0 to a452f12 Compare August 23, 2024 21:14
@pajod pajod force-pushed the patch-fcntl-FD_CLOEXEC-ioctl-FIOCLEX branch from f8bda39 to 0d03ef2 Compare September 8, 2024 21:45
@pajod pajod marked this pull request as ready for review September 9, 2024 10:57
@pajod pajod force-pushed the patch-fcntl-FD_CLOEXEC-ioctl-FIOCLEX branch from 0d03ef2 to 02f9d03 Compare November 27, 2024 21:38
gunicorn invocation now matches between the two similar tests
nginx needs worker=off to shut down when signaled
gevent worker on OmniOS py3.12/gcc14 fails with:
AttributeError: module 'threading' has no attribute 'get_native_id'

skip slow benchmark on PyPy
on gevent 24.10.1 to 24.11.1
gevent worker on OmniOS py3.12/gcc14 fails with:
AttributeError: module 'threading' has no attribute 'get_native_id'
see gevent/gevent#2053
@pajod pajod force-pushed the patch-fcntl-FD_CLOEXEC-ioctl-FIOCLEX branch from 6402a5c to ff48a1f Compare November 30, 2024 22:46
@benoitc benoitc added the Testing label Dec 1, 2024
@benoitc
Copy link
Owner

benoitc commented Dec 1, 2024

i will look at this branch but I am questionning the goal pursued there. One of the point of keeping low level though is to not depends on how one external dev think things should work. Knowing and controlling low level access allows us to understand and track more accurately some issues. Thoughts?

@pajod
Copy link
Contributor Author

pajod commented Dec 1, 2024

Goals:

  1. simplify code that was not serving a role in understanding low-level issues
  2. make tests work on windows (this only eventually, after a few other patches)
  3. start running some end-to-end testing in CI, well beyond individual unit tests, that helps with understanding low-level issues

The first goal is to remove code that is difficult to diagnose for purely historical, no longer applicable, reasons. cPython did not offer these shorthands back then. Now it does. We never gained or lost any control over that particular low-level API, because either it will do what we expect, or worse: it will pretend to offer matching API, but with differences only observable elsewhere.

For those cases, I am all in favor of adding a generous amount of verbosity in clearly platform-specific branches e.g. extend error handlers: pajod@7475a75

not depend on how one external dev think things should work

Specifically, I would like to stop caring whether Windows thinks one should call incompatible ioctlsocket(fd, FIONBIO, ..) and SetNamedPipeHandleState(fd_to_handle(fd), old_mode | PIPE_NOWAIT, ..) calls for essentially the "same" behavior.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Missing test cases for workers
2 participants