Running several verdi run processes independently crashes due to UniquenessError when creating the auto groups

[lekah]

I found a case of a calculation going into SUBMISSIONFAILED with the following log-message:

*** 1739620 [Li27Na27C27O81-1000-branching-19-NVE-5-0]: SUBMISSIONFAILED
*** Scheduler output: N/A
*** Scheduler errors: N/A
*** 1 LOG MESSAGES:
+-> ERROR at 2017-12-17 20:29:05.260324+00:00
| Submission of calc 1739620 failed, check also the log file! Traceback: Traceback (most recent call last):
| File "/home/kahle/git/AiiDA/aiida_core-screening/aiida/daemon/execmanager.py", line 623, in submit_calc
| remotedata.store()
| File "/home/kahle/git/AiiDA/aiida_core-screening/aiida/orm/implementation/general/node.py", line 1571, in store
| name=group_name, type_string=VERDIAUTOGROUP_TYPE)[0]
| File "/home/kahle/git/AiiDA/aiida_core-screening/aiida/orm/implementation/general/group.py", line 149, in get_or_create
| bla = cls(*args, **kwargs).store(), True
| File "/home/kahle/git/AiiDA/aiida_core-screening/aiida/orm/implementation/django/group.py", line 127, in store
| raise UniquenessError("A group with the same name (and of the "
| UniquenessError: A group with the same name (and of the same type) already exists, unable to store

I suspect that the get_or_create method of the Group is not safe for multi-threaded/multi-processing environment, because the underlying implementation is not an 'INSERT IF' construct, but first a query, and then the storage, which can result in the Uniqueness being broken.
We need to make the get_or_create method threadsafe.
I also, what is the point of grouping submitted calculations? Is that a necessary feature?

[sphuber]

The problem is that @lekah was launching his calculations through a function that was called with verdi run. The verdi run command has a -g option that is enabled by default and will use the Autogroup class to automatically assign all the created nodes in the execution to this group. If no specific value is passed for the group name, a name is generated based on the current time to the nearest second, e.g. Verdi autogroup on 2017-11-06 12:26:18. If the verdi run call is done in parallel, multiple processes may incur the autogroup call within the same second and both attempt to create the same group, leading to the UniquenessError.

The core problem is that the Group.get_or_create function is not thread-safe.
However, I also think the choice of making the -g option enable by default for verdi run is an odd choice. I never even knew this existed and I have tons of useless groups created in my database that you can only see if you actually query the database directly.

[giovannipizzi]

Probably the solution is to document that AiiDA is not 'thread-safe'

[sphuber]

@giovannipizzi do you have any suggestion on where to put this in the documentation?

[giovannipizzi]

Mmm... I'm not sure... I think this goes in hand with all the discussion we had about checking all other thread/process-safety (e.g. when storing attributes, extras, ...). I don't remember if we said this can happen for the 1.0?

Anyway, thinking back to this after a few months, I think that if we manage to fix the get_or_create function to avoid problems, it's the best solution (actually, in this case, it should be as simple as try/catching the UniquenessError in the group creation [for the autogroup only]) and if this happens, retry to create a new group with -NUM appended, incrementing NUM until when it succeeds (maybe with a large safety max, e.g. a few thousands?) [we need also to understand how to test it, though, an integration test is complex, but maybe we can think to a unit test where the groups are already there and we see if the right group name is generated].

[giovannipizzi]

Small script to create 30 nodes "quickly":

#!/usr/bin/env runaiida
import os
import time
from datetime import datetime
from aiida.orm import Data

N = 30

current_pid = os.getpid()

print(current_pid)

nodes = []
for i in range(N):
    time.sleep(0.1)
    data = Data().store()
    nodes.append(data.pk)

print(nodes)

You can save the script above in a file spawn.py and then run it from bash as

(./spawn.py & ) ; ( ./spawn.py & )

so that they run in parallel. The second will crash with a traceback similar to the one above.
We can fix this, and the check how to make a test.