110 - Don't stop all tasks when experiencing unexpected worker deaths #134
Conversation
mpire/async_result.py
Outdated
| MAIN = 1 | ||
| INIT = 2 | ||
| MAP = 3 | ||
| EXIT = 4 | ||
| APPLY = 5 |
There was a problem hiding this comment.
Instead of explicitly setting enum values, you could also use enum.auto().
| def __init__( | ||
| self, | ||
| cache: Dict, | ||
| callback: Optional[Callable], | ||
| error_callback: Optional[Callable], | ||
| job_id: Optional[int] = None, | ||
| delete_from_cache: bool = True, | ||
| timeout: Optional[float] = None, | ||
| ) -> None: |
There was a problem hiding this comment.
You've started applying black?
There was a problem hiding this comment.
Yeah, I'm applying it to some files here and there. Will do a full black conversion later.
Also planning on introducing the other precommit checks, toml file etc. But that will take some time
There was a problem hiding this comment.
Well, I can't honestly say the changes it made in this PR improved it. I found it nicer to read before. I think it'll take some fancier automation to replace your sense of code style ;)
There was a problem hiding this comment.
Haha, thnx :D Yeah, I'm also not in agreement with all the choices black makes, but it's so much easier to simply hit the format code shortcut in the editor and be done with it
sybrenjansen
deleted the
110-_unexpected_death_handler-dont-stop-all-tasks-just-the-task-that-failed
branch
In the case of an unexpected worker death (e.g., OOM errors) and the worker was working on an apply task, the worker will now be restarted and the other workers will continue their work. The task that caused the death will be set to failed
Fixes #110