Make the execmanager.submit_calculation idempotent'ish
#3188
Conversation
The `submit_calculation` is responsible for submitting a job, that was already uploaded to the remote machine, to the scheduler. There was a risk of the daemon submitting the same job twice. If the first time around it successfully submitted the job and set the job id as an attribute on the node, but then was interrupted (for example due to a shutdown) before it could transition the `CalcJob` to the `UPDATE` transport task, once reloaded the command would be executed again. This would not complain and simply submit the job again to the scheduler. This is both a waste of resources and can cause various complications. It is impossible to make the function fully idempotent, but by checking if the job id attribute is already set for the node at the beginning and returning it should minimize the risk of double submission.
sphuber
force-pushed
the
fix_3183_submit_calculation_idempotency
branch
from
b44b0da to
caceba0
Compare
There was a problem hiding this comment.
Very good!
Another thing to check when we address #2496 is that the set_job_id call is atomic and stores asap.
as you say this is not enough if the daemon gets shut down before that call but after submission.
The only option to make this really idempotent is to delegate the check to the scheduler, but this is beyond the scope of this PR here.
Eg have a wrapper for squeue that submits only once per folder (e. g. storing the jobid on a file in the folder, say _aiida_jobid.txt, and before submission it checks if it exists - if it's there it returns it, otherwise it proxies to squeue.
Not for now, but would it be possible/easy to get from slurm all the job id's you have submitted, say, in the past 1 week, and check if any of them is unknown to AiiDA? (e. g. meaning that even with this fix we would have run the simulation twice)?
aybe this can be tested also after this PR is merged.
|
see e.g. slurm commands here https://ubccr.freshdesk.com/support/solutions/articles/5000686909-how-to-retrieve-job-history-and-accounting |
|
Should be easy enough. To get a unique list of all job ids for given user since given data: this can then be compared with the job ids stored in the attributes. |
) The `submit_calculation` is responsible for submitting a job, that was already uploaded to the remote machine, to the scheduler. There was a risk of the daemon submitting the same job twice. If the first time around it successfully submitted the job and set the job id as an attribute on the node, but then was interrupted (for example due to a shutdown) before it could transition the `CalcJob` to the `UPDATE` transport task, once reloaded the command would be executed again. This would not complain and simply submit the job again to the scheduler. This is both a waste of resources and can cause various complications. It is impossible to make the function fully idempotent, but by checking if the job id attribute is already set for the node at the beginning and returning it should minimize the risk of double submission.
) The `submit_calculation` is responsible for submitting a job, that was already uploaded to the remote machine, to the scheduler. There was a risk of the daemon submitting the same job twice. If the first time around it successfully submitted the job and set the job id as an attribute on the node, but then was interrupted (for example due to a shutdown) before it could transition the `CalcJob` to the `UPDATE` transport task, once reloaded the command would be executed again. This would not complain and simply submit the job again to the scheduler. This is both a waste of resources and can cause various complications. It is impossible to make the function fully idempotent, but by checking if the job id attribute is already set for the node at the beginning and returning it should minimize the risk of double submission.
Fixes #3183
The
submit_calculationis responsible for submitting a job, that wasalready uploaded to the remote machine, to the scheduler. There was a
risk of the daemon submitting the same job twice. If the first time
around it successfully submitted the job and set the job id as an
attribute on the node, but then was interrupted (for example due to a
shutdown) before it could transition the
CalcJobto theUPDATEtransport task, once reloaded the command would be executed again. This
would not complain and simply submit the job again to the scheduler.
This is both a waste of resources and can cause various complications.
It is impossible to make the function fully idempotent, but by checking
if the job id attribute is already set for the node at the beginning and
returning it should minimize the risk of double submission.