orphaned postgres processes after running tests

[ltalirz]

Running aiida tests can lead to orphaned postgres servers.

If at some point during testing you encounter errors like this one:

IOError: FATAL:  could not create shared memory segment: No space left on device
DETAIL:  Failed system call was shmget(key=1, size=56, 03600).
HINT:  This error does *not* mean that you have run out of disk space.  It occurs either if all available shared memory IDs have been taken, in which case you need to raise the SHMMNI parameter in your kernel, or because the system's overall limit for shared memory has been reached.
	The PostgreSQL documentation contains more information about shared memory configuration.
child process exited with exit code 1

this may be because there are a number of orphaned postgres servers running:

ps -ef | grep postgres
...
  502  7662     1   0  5:45PM ttys009    0:00.08 /opt/local/lib/postgresql96/bin/postgres -D /var/folders/8y/7c7qkxcx5vxb749rt3sw5sh40000gp/T/tmpAp52Z0/data -F -d 1 -p 55023 -c logging_collector=off -N 11 -k /var/folders/8y/7c7qkxcx5vxb749rt3sw5sh40000gp/T/tmpAp52Z0/tmp
  502  7700     1   0  5:45PM ttys009    0:00.08 /opt/local/lib/postgresql96/bin/postgres -D /var/folders/8y/7c7qkxcx5vxb749rt3sw5sh40000gp/T/tmpk6mgLM/data -F -d 1 -p 55034 -c logging_collector=off -N 11 -k /var/folders/8y/7c7qkxcx5vxb749rt3sw5sh40000gp/T/tmpk6mgLM/tmp
  502  7734     1   0  5:45PM ttys009    0:00.06 /opt/local/lib/postgresql96/bin/postgres -D /var/folders/8y/7c7qkxcx5vxb749rt3sw5sh40000gp/T/tmp_79xLj/data -F -d 1 -p 55044 -c logging_collector=off -N 11 -k /var/folders/8y/7c7qkxcx5vxb749rt3sw5sh40000gp/T/tmp_79xLj/tmp
  502  7771     1   0  5:45PM ttys009    0:00.08 /opt/local/lib/postgresql96/bin/postgres -D /var/folders/8y/7c7qkxcx5vxb749rt3sw5sh40000gp/T/tmpuHaxqL/data -F -d 1 -p 55047 -c logging_collector=off -N 11 -k /var/folders/8y/7c7qkxcx5vxb749rt3sw5sh40000gp/T/tmpuHaxqL/tmp
  502  7801     1   0  5:45PM ttys009    0:00.06 /opt/local/lib/postgresql96/bin/postgres -D /var/folders/8y/7c7qkxcx5vxb749rt3sw5sh40000gp/T/tmpoq1qcc/data -F -d 1 -p 55055 -c logging_collector=off -N 11 -k /var/folders/8y/7c7qkxcx5vxb749rt3sw5sh40000gp/T/tmpoq1qcc/tmp
  502  7827     1   0  5:45PM ttys009    0:00.06 /opt/local/lib/postgresql96/bin/postgres -D /var/folders/8y/7c7qkxcx5vxb749rt3sw5sh40000gp/T/tmpYHRqRs/data -F -d 1 -p 55058 -c logging_collector=off -N 11 -k /var/folders/8y/7c7qkxcx5vxb749rt3sw5sh40000gp/T/tmpYHRqRs/tmp
  502 15456     1   0  6:05PM ttys009    0:00.04 /opt/local/lib/postgresql96/bin/postgres -D /var/folders/8y/7c7qkxcx5vxb749rt3sw5sh40000gp/T/tmpnEwsEM/data -F -d 1 -p 55554 -c logging_collector=off -N 11 -k /var/folders/8y/7c7qkxcx5vxb749rt3sw5sh40000gp/T/tmpnEwsEM/tmp
  502 15491     1   0  6:05PM ttys009    0:00.05 /opt/local/lib/postgresql96/bin/postgres -D /var/folders/8y/7c7qkxcx5vxb749rt3sw5sh40000gp/T/tmp83IZ1V/data -F -d 1 -p 55559 -c logging_collector=off -N 11 -k /var/folders/8y/7c7qkxcx5vxb749rt3sw5sh40000gp/T/tmp83IZ1V/tmp
  502 15521     1   0  6:06PM ttys009    0:00.04 /opt/local/lib/postgresql96/bin/postgres -D /var/folders/8y/7c7qkxcx5vxb749rt3sw5sh40000gp/T/tmp4tBSZ6/data -F -d 1 -p 55562 -c logging_collector=off -N 11 -k /var/folders/8y/7c7qkxcx5vxb749rt3sw5sh40000gp/T/tmp4tBSZ6/tmp
  502 15584     1   0  6:06PM ttys009    0:00.04 /opt/local/lib/postgresql96/bin/postgres -D /var/folders/8y/7c7qkxcx5vxb749rt3sw5sh40000gp/T/tmpzzh_Y1/data -F -d 1 -p 55565 -c logging_collector=off -N 11 -k /var/folders/8y/7c7qkxcx5vxb749rt3sw5sh40000gp/T/tmpzzh_Y1/tmp
  502 15612     1   0  6:06PM ttys009    0:00.05 /opt/local/lib/postgresql96/bin/postgres -D /var/folders/8y/7c7qkxcx5vxb749rt3sw5sh40000gp/T/tmpNSotYs/data -F -d 1 -p 55569 -c logging_collector=off -N 11 -k /var/folders/8y/7c7qkxcx5vxb749rt3sw5sh40000gp/T/tmpNSotYs/tmp
  502 15641     1   0  6:06PM ttys009    0:00.04 /opt/local/lib/postgresql96/bin/postgres -D /var/folders/8y/7c7qkxcx5vxb749rt3sw5sh40000gp/T/tmppOFRpd/data -F -d 1 -p 55573 -c logging_collector=off -N 11 -k /var/folders/8y/7c7qkxcx5vxb749rt3sw5sh40000gp/T/tmppOFRpd/tmp
  502 16264     1   0  6:07PM ttys009    0:00.08 /opt/local/lib/postgresql96/bin/postgres -D /var/folders/8y/7c7qkxcx5vxb749rt3sw5sh40000gp/T/tmp1uMS7v/data -F -d 1 -p 55610 -c logging_collector=off -N 11 -k /var/folders/8y/7c7qkxcx5vxb749rt3sw5sh40000gp/T/tmp1uMS7v/tmp
  502 16318     1   0  6:07PM ttys009    0:00.08 /opt/local/lib/postgresql96/bin/postgres -D /var/folders/8y/7c7qkxcx5vxb749rt3sw5sh40000gp/T/tmpkGPIPN/data -F -d 1 -p 55623 -c logging_collector=off -N 11 -k /var/folders/8y/7c7qkxcx5vxb749rt3sw5sh40000gp/T/tmpkGPIPN/tmp
  502 16361     1   0  6:07PM ttys009    0:00.06 /opt/local/lib/postgresql96/bin/postgres -D /var/folders/8y/7c7qkxcx5vxb749rt3sw5sh40000gp/T/tmpB58fqI/data -F -d 1 -p 55634 -c logging_collector=off -N 11 -k /var/folders/8y/7c7qkxcx5vxb749rt3sw5sh40000gp/T/tmpB58fqI/tmp
  502 16396     1   0  6:07PM ttys009    0:00.05 /opt/local/lib/postgresql96/bin/postgres -D /var/folders/8y/7c7qkxcx5vxb749rt3sw5sh40000gp/T/tmpiC1qss/data -F -d 1 -p 55639 -c logging_collector=off -N 11 -k /var/folders/8y/7c7qkxcx5vxb749rt3sw5sh40000gp/T/tmpiC1qss/tmp
  502 16424     1   0  6:07PM ttys009    0:00.04 /opt/local/lib/postgresql96/bin/postgres -D /var/folders/8y/7c7qkxcx5vxb749rt3sw5sh40000gp/T/tmpW50bXh/data -F -d 1 -p 55644 -c logging_collector=off -N 11 -k /var/folders/8y/7c7qkxcx5vxb749rt3sw5sh40000gp/T/tmpW50bXh/tmp
  502 16457     1   0  6:07PM ttys009    0:00.10 /opt/local/lib/postgresql96/bin/postgres -D /var/folders/8y/7c7qkxcx5vxb749rt3sw5sh40000gp/T/tmprONbxG/data -F -d 1 -p 55648 -c logging_collector=off -N 11 -k /var/folders/8y/7c7qkxcx5vxb749rt3sw5sh40000gp/T/tmprONbxG/tmp
  502 17944     1   0  6:12PM ttys009    0:00.07 /opt/local/lib/postgresql96/bin/postgres -D /var/folders/8y/7c7qkxcx5vxb749rt3sw5sh40000gp/T/tmp8jOFmy/data -F -d 1 -p 55757 -c logging_collector=off -N 11 -k /var/folders/8y/7c7qkxcx5vxb749rt3sw5sh40000gp/T/tmp8jOFmy/tmp
  502 17992     1   0  6:12PM ttys009    0:00.07 /opt/local/lib/postgresql96/bin/postgres -D /var/folders/8y/7c7qkxcx5vxb749rt3sw5sh40000gp/T/tmpuGwNYJ/data -F -d 1 -p 55770 -c logging_collector=off -N 11 -k /var/folders/8y/7c7qkxcx5vxb749rt3sw5sh40000gp/T/tmpuGwNYJ/tmp
  502 18031     1   0  6:12PM ttys009    0:00.06 /opt/local/lib/postgresql96/bin/postgres -D /var/folders/8y/7c7qkxcx5vxb749rt3sw5sh40000gp/T/tmpCf56tV/data -F -d 1 -p 55781 -c logging_collector=off -N 11 -k /var/folders/8y/7c7qkxcx5vxb749rt3sw5sh40000gp/T/tmpCf56tV/tmp
  502 18069     1   0  6:12PM ttys009    0:00.06 /opt/local/lib/postgresql96/bin/postgres -D /var/folders/8y/7c7qkxcx5vxb749rt3sw5sh40000gp/T/tmpzwG2O4/data -F -d 1 -p 55785 -c logging_collector=off -N 11 -k /var/folders/8y/7c7qkxcx5vxb749rt3sw5sh40000gp/T/tmpzwG2O4/tmp
  502 21552 72894   0  6:21PM ttys009    0:00.00 grep --color=auto postgr

I've encountered this previously but I may not have opened an issue (at least could not find it now).

[ltalirz]

Turns out the problem is here, which is part of verdi devel tests db.manage.external.postgres
https://github.com/aiidateam/aiida_core/blob/f8c0191e9e129f6c54bf9a25480f911b4d93b13d/aiida/backends/tests/manage/external/test_postgres.py#L33

This test class is creating a new database cluster for each test - but not removing them after the tests are finished!

A simple

def tearDown():
    self.pg_test.close()

solves the issue (will be included in #2409)

[sphuber]

Fixed in PR #2487