-
-
Notifications
You must be signed in to change notification settings - Fork 30.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
multiprocessor spawn #84287
Comments
MacOS Catalina 10.15.3 and 10.15.4. Python-3.8.2 (also tested with 3.7.7, which confirmed the problem being in the fix described in https://bugs.python.org/issue33725. Trying to use "multiprocessor" with Python-3.8 and with the new default of Not doing join() leads to consistent crashes, like described here https://bugs.python.org/issue33725#msg365249 Adding p.join() immediately after p.start() seems to work, but increases the total run-time by factor between two and four, user time by factor of five, and system time by factor of ten. Occasionally even with p.join() I'm getting some processes crashing like shown in https://bugs.python.org/issue33725#msg365249. I found two workarounds:
If anybody cares to dig deeper into this problem, I'd be happy to provide whatever information that could be helpful. Here's the sample code (again):
Here's the timing:
If I comment out the join() and uncomment set_start_method('fork'), the timing is
You can observe the difference. Here's the timing if I don't bother with either join() or set_start_method(), but import "multiprocess" instead:
Also, on a weaker machine with only 4 cores (rather than 20 that ran the above example), the instability of the "multiprocessor"-based code shows:
Observe how one process out of four got nothing from the queue. With "multiprocess" the code runs like a clockwork - each process gets exactly 1/N of the queue:
I think that the best course for "multiprocessor" would be reverting the default to 'fork'. It also looks like for the users the best course would be switching to "multiprocess". |
@Mouse, using "multiprocess" instead of "multiprocessing" will not work if you're passing a class that inherits from ABC. "dill" is one of "multiprocess"'s dependencies and "dill" can't pickle an _abc_data object (uqfoundation/dill#332) |
I'm closing this issue, the switch from "fork" to "spawn" on macOS was done to avoid crashes in code that links to higher-level APIs on macOS, including some bits of the stdlib (in particular the _scproxy extension used by urllib). |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: