Engine: Add the wait argument to submit
#6155
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
sphuber
force-pushed
the
feature/submit-wait
branch
2 times, most recently
from
aec0384 to
3352c94
Compare
For demos and tutorials, often in interactive notebooks, it is often preferred to use `run` instead of `submit` because in this way the cell will block until the process is done. The cell blocking will signal to the user that the process is still running and as soon as it returns it is immediately clear that the results are ready. With `submit` the cell returns immediately, but the user will now have to resort to manually checking when the process is done. Solutions are to instruct the user to call `verdi process list` manually (which they will have to do repeatedly) or implement some automated loop that checks for the process to terminate. However, using `run` has downsides as well, most notably that the process will be lost if the notebook gets disconnected. For processes that are expected to run longer, this can be really problematic, and so `submit` will have to be used regardless. Here, the `wait` argument is added to `submit`. Set to `False` by default to keep current behavior, when set to `True`, the function will mimic the behavior of `run` and only return when the process has terminated at which point the node is returned. A `REPORT` log message is emitted each time the state of the process is checked in intervals of `wait_interval`.
sphuber
force-pushed
the
feature/submit-wait
branch
from
3352c94 to
406d103
Compare
sphuber
force-pushed
the
feature/submit-wait
branch
from
406d103 to
b6f7e2d
Compare
The recent addition of the `wait` argument to the `submit` function allows a user to submit process the daemon, while still have the function block until the process is terminated, as a call to `run` would do. This can be useful in interactive tutorials and demos where the code should not avance until the process is done, but one still wants to benefits of having the daemon run the process. The downside of this approach is that it only allows to submit and wait for a single process at a time. Here the `await_processes` function is added. It takes a list of process nodes and will wait in a loop for all of them reach a terminal state. The time between iterations can be controlled by the `wait_interval` argument.
mbercx
previously approved these changes
Oct 27, 2023
| One could of course also use ``run`` (see below), but then the process would be lost if the interpreter gets accidentally shut down. | ||
| By using ``submit``, the process is run by the daemon which takes care of saving checkpoints so it can always be restarted in case of problems. | ||
| If you need to launch multiple processes in parallel and want to wait for all of them to be finished, simply use ``submit`` with the default ``wait=False`` and collect the returned nodes in a list. | ||
| You can then pass them to :func:`aiida.engine.launch.await_processes` which will return once all processes have terminated:: |
There was a problem hiding this comment.
Suggested change
| You can then pass them to :func:`aiida.engine.launch.await_processes` which will return once all processes have terminated:: | |
| You can then pass them to :func:`aiida.engine.launch.await_processes` which will return once all processes have terminated: |
I think the extra colon is still causing trouble here.
There was a problem hiding this comment.
Thanks for catching that. I tested the docs locally and it passed, but didn't bother to look at the actual output. Funnily enough, with just a single colon VSCode actually has incorrect syntax highlighting and lights up the whole phrase starting from :func: onwards. This was fixed by the double colon, but clearly that is not what restructured text wants.
I also noted that I forgot to update sleep_interval to wait_interval which I corrected now
sphuber
force-pushed
the
feature/submit-wait
branch
from
b6f7e2d to
f91aa09
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
For demos and tutorials, often in interactive notebooks, it is often preferred to use
runinstead ofsubmitbecause in this way the cell will block until the process is done. The cell blocking will signal to the user that the process is still running and as soon as it returns it is immediately clear that the results are ready. Withsubmitthe cell returns immediately, but the user will now have to resort to manually checking when the process is done. Solutions are to instruct the user to callverdi process listmanually (which they will have to do repeatedly) or implement some automated loop that checks for the process to terminate.However, using
runhas downsides as well, most notably that the process will be lost if the notebook gets disconnected. For processes that are expected to run longer, this can be really problematic, and sosubmitwill have to be used regardless.Here, the
waitargument is added tosubmit. Set toFalseby default to keep current behavior, when set toTrue, the function will mimic the behavior ofrunand only return when the process has terminated at which point the node is returned. AREPORTlog message is emitted each time the state of the process is checked in intervals ofwait_interval.