Monitor Nomad Health

[mpass99]

In the context of #390 we remove misleading Sentry events. Those events also used to report errors within Nomad. Besides that, we may only notice a bad health due to an Ansible deployment if the host is not reachable at all.

With this issue, we want to create a monitoring for the Nomad hosts. This monitoring might include the information of sudo nomad node status such as the Status or the Eligibility of the agents.

[MrSerth]

Besides checking the Nomad health directly through an external tool (such as Icinga), we could also check it indirectly through Poseidon. This would ensure that Poseidon is the one place to contact about the health of code executions, and would further scale with additional execution environments (such as serverless solutions).

Therefore, it would be nice if we could add a dedicated route to Poseidon allowing to check the health of all environments (such as Nomad, AWS, ...). We decided to keep this route separated from the existing health check for an independent check, e.g., with a different interval.

[MrSerth]

This ticket might become obsolete by #471 and an additional health monitoring in Poseidon (see #470 (comment)).