-
-
Notifications
You must be signed in to change notification settings - Fork 31.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
bpo-35621: Support running subprocesses in asyncio when loop is executed in non-main thread #13630
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is it possible to make MultiChildWatcher to work like FastChildWatcher? Can we just fix them without introducing a new class? I think we discussed these options but I don't remember the resolution.
Sorry, no.
Currently we have O(N)
That's how asyncio works now and from the very beginning. I'm not very comfortable with changing policy on-demand initialization too much now. Calling |
|
I've quickly prototyped this: #13656 |
How about we don't allow users to use subprocess APIs from non-main threads if we don't have a working child watcher ready? |
I suspect that
I can implement
Also, I don't like to create a waiting thread per subprocess. I suspect that this approach doesn't scale well for 10,000 subprocesses. I don't want to apply this change in the last day before the feature freeze. We can return to a discussion about subprocess support redesign in Python 3.9 |
How about we rename ThreadedChildWatcher to DefaultChildWatcher? My reasoning: if it's called the latter, it could implement different watching strategy based on OS or Python version. On macOS we could use kwueue, on linux threads etc. Alternatively, we can keep this as is, implement KQueue watcher (in 3.9) and let asyncio decide which one should be the default at import time. Maybe this option is better and less confusing. Thoughts? |
Let's keep it named ThreadedChildWatcher :) |
Codecov Report
@@ Coverage Diff @@
## master #13630 +/- ##
==========================================
+ Coverage 82.76% 82.77% +<.01%
==========================================
Files 1843 1843
Lines 559912 559893 -19
Branches 41403 41431 +28
==========================================
+ Hits 463417 463457 +40
+ Misses 87421 87350 -71
- Partials 9074 9086 +12
Continue to review full report at Codecov.
|
@asvetlov: Please replace |
|
…is executed in non-main thread (python#13630)"
…is executed in non-main thread (#13630)" (GH-13793) https://bugs.python.org/issue35621
…en loop is executed in non-main thread (python#13630)" (pythonGH-13793)" This reverts commit 9535aff.
…ted in non-main thread (python#13630)
…is executed in non-main thread (python#13630)" (pythonGH-13793) https://bugs.python.org/issue35621
Implementation adds
MultiLoopChildWatcher
and uses it by default.https://bugs.python.org/issue35621