add verdi status command

[ltalirz]

• fix implement a verdi status command #2409
• fix orphaned postgres processes after running tests #2519

Adds a verdi status command that shows the status of the different AiiDa services.

• show profile status
• show file repo status
• show postgres status
• show rabbitmq status
• show daemon status

I'm sure it can still be improved and made more robust, but it's a start.

To do:

• add documentation
• add tests
• remove dependency on load_dbenv. Currently, the command loads the dbenv because get_communicator requires it. It is not clear why this is necessary, perhaps @sphuber can clarify this (and possibly provide a workaround). I wonder - should aiida.work.rmq actually be moved to aiida.manage.external?
• open an issue for further improvements:
  • When the file repository does not exist, AiiDA currently creates a new folder automatically and without any warning. This makes the current "file repository status" rather meaningless, since it only checks whether the folder exists. I would suggest to provide a warning, when AiiDA has to create it from scratch and to add some info to the status, e.g. whether the folder is empty or not (open to suggestions)
  • could consider adding version information that helps as reports on mailing lists (e.g. aiida version, postgres version, rmq version)
• use different symbols to indicate UP/DOWN so that it can be copy-pasted in forums/github without losing meaning

Latest version

$ verdi status
  ✓ profile:   On profile test_qb
  ✓ repository: /Users/leopold/Personal/Postdoc-MARVEL/aiida_folders/aiida_rmq/.aiida/repository/test_qb
  ✓ postgres:  Connected to postgres@localhost:5432
  ✓ rabbitmq:  Connected to amqp://127.0.0.1?heartbeat=600
  ✗ daemon:    The daemon is not running

[coveralls]

Coverage increased (+0.04%) to 69.766% when pulling 6202948 on ltalirz:issue_2409_verdi_status into f8c0191 on aiidateam:provenance_redesign.

[giovannipizzi]

Cool! Do you plan to fix all the tick boxes before merging? In this case, can you add [WIP] to the title?

[ltalirz]

@giovannipizzi Done. Can you perhaps comment on the last todo item?

[giovannipizzi]

I suggest to only check existence for now, and mention this in the issue about improving the file repository - I don't think it makes sense to spend toouch effort there now, and additional checks might not be particularly useful at this stage.

[ltalirz]

Added test, which passes on my machine while still showing the following error:

vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
>>> Tests for django db application
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.Exception in thread Communications thread for 'RMQCommunicator (amqp://guest:********@127.0.0.1:5672//)':
Traceback (most recent call last):
  File "/Users/leopold/Applications/miniconda3/envs/aiida_rmq/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/Users/leopold/Applications/miniconda3/envs/aiida_rmq/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/Users/leopold/Applications/miniconda3/envs/aiida_rmq/lib/python2.7/site-packages/kiwipy/rmq/communicator.py", line 543, in run_loop
    self._loop.start()
  File "/Users/leopold/Applications/miniconda3/envs/aiida_rmq/lib/python2.7/site-packages/tornado/ioloop.py", line 863, in start
    event_pairs = self._impl.poll(poll_timeout)
  File "/Users/leopold/Applications/miniconda3/envs/aiida_rmq/lib/python2.7/site-packages/tornado/platform/kqueue.py", line 66, in poll
    kevents = self._kqueue.control(None, 1000, timeout)
OSError: [Errno 9] Bad file descriptor

@sphuber Do you have an idea what this means?
I don't get this error when I run verdi status directly.

[sphuber]

@ltalirz no unfortunately, I have never seen that exception before

[sphuber]

@ltalirz would this be ready for review?

[ltalirz]

Happy for your feedback if you find the time.
I don't plan any significant changes but I still need to figure out the problem with the test mentioned above.

Another problem is that, even with Capturing, I still get prints of exceptions printed to the terminal if there are problems with rabbitmq.

[sphuber]

Probably just because it is writing to stderr and you did not pass capture_stderr=True to the constructor of the Capturing class. Give that a try

[ltalirz]

@sphuber both issues should be fixed now.
Maybe give the command a try yourself.

[sphuber]

Just did and got the following:

  ✓ profile:   On profile django
  ✓ repository: /repo/aiida_prov/django
  ✗ postgres:  Error connecting to localhost:5432
'user'
  ✓ rabbitmq:  Connected to amqp://127.0.0.1?heartbeat=600
  ✗ daemon:    The daemon is not running

Clearly there is a problem with the call to postgres, but that should not be the case. Will see the reason why.

Then regarding layout, it would be nice if we can line the values out on the same column. Now it seems that there is a space in front of the repository path

[sphuber]

Problem for postgres failing is because of the line dbinfo['user'] but in my case the dbinfo does not have the user key. It seems this is the case because it is not being set in the Postgres.from_profile method and the constructor does not allow to set it. I changed this so the KeyError disappears, but then I still get a failure:

✗ postgres:  Unable to connect to aiida@localhost:5432

Even though it should be able to connect just fine.

[ltalirz]

Thanks, yes I thought that determine_setup() is not ideal.
It might try doing some stuff that actually requires sudo on your machine.
I'll look into it.

[sphuber]

@ltalirz just a quick FYI: each time you push a commit to an open PR will trigger a new build, blocking all other builds for the aiidateam/aiida-core repo. You can still push multiple commits, but in that case it would be nice to cancel the builds from the commits before, because they will be overridden anyway. I just canceled all but the last build for this PR to clear the queue a bit

[ltalirz]

closing this until tests pass

[sphuber]

I tested it locally on Ubuntu 18.04 and it seems to be working. This is the scenario where all services are up. Only few minor comments remain

[sphuber]

Thanks a lot @ltalirz