Fix bug in process control in direct scheduler (#2721)

The direct scheduler uses `ps` for checking whether a command is
running. However, `ps` does not show processes without a controlling
terminal by default - for this you need to add the -x option:

  -x      When displaying processes matched by other options, include pro-
           cesses which do not have a controlling terminal.  This is the
           opposite of the -X option.  If both -X and -x are specified in
           the same command, then ps will use the one which was specified
           last.

Since this flag was not there, the engine thought calculations were finished
when they were not and so the engine started retrieving files prematurely.
Adding the flag solves this issue.

aiida/schedulers/plugins/direct.py

@@ -93,7 +93,11 @@ def _get_joblist_command(self, jobs=None, user=None):
         TODO: in the case of job arrays, decide what to do (i.e., if we want
               to pass the -t options to list each subjob).
         """
-        command = 'ps -o pid,stat,user,time'
+        # Using subprocess.Popen with `preexec_fn=os.setsid` (as done in both local and ssh transport) results in
+        # processes without a controlling terminal.
+        # The -x option tells ps to include processes which do not have a controlling terminal, which would not be
+        # listed otherwise (leading the direct scheduler to conclude that the process already completed).
+        command = 'ps -xo pid,stat,user,time'
 
         if jobs:
             if isinstance(jobs, six.string_types):

docs/source/developer_guide/core/internals.rst

@@ -460,3 +460,8 @@ You should be aware of this while developing code which runs in the daemon. In p
 
     print(subprocess.check_output('sleep 3; echo bar', preexec_fn=os.setsid))
 
+.. note::
+
+    When dropping python 2.7 support, ``preexec_fn=os.setsid`` should be replaced
+    by the thread safe ``start_new_session=True`` introduced in python 3.2.
+