Alert manager based utilization enhancement

[suiguoxin]

This PR is for the first part of issue #4789

• Alert-manager: Kill low-gpu-utilization jobs, tag abnormal jobs
  • add virtual cluster info in job-exporter
  • config monitor rules in prometheus
  • send action request through webhook
  • job-handler: deal with webhook request & redirect to RestServer
  • realize customized SMTP service in alert-handler, send alert email to user when possible, change email template to ejs
  • document how to customize alerts/actions
• Job tags:
  • DB: job-tag table
  • RestServer:
    • getJobList : filter by tag
    • getJobDetails : with tag info
    • executionJob : tag / untag
• WebPortal abnormal jobs: refactor with tag filter

Realization Details:

• add vc info in job-exporter
• set Prometheus to monitor jobs in certain virtual clusters with low gpu usage, send email and call webhook (alert-handler)
• add alert-handler as a container in alert-manager, transfer webhook call to REST-SERVER to stop job
• Customized Configuration (bearer token / alert config) should be set in services-configuration.yaml.template

[coveralls]

Coverage remained the same at 34.383% when pulling 27408d4 on suiguoxin:prometheus into 9755553 on microsoft:master.

[Binyang2014]

We discussed about move tags under tag-jobs. Can we achieve this?

[suiguoxin]

We discussed about move tags under tag-jobs. Can we achieve this?

It is legal to mix dict / string in the same list in yaml, but it becomes difficult to check if tag-jobs is one of the available actions in the template. So I suggest to keep the current schema.

[Binyang2014]

Thanks for this enhance.

[suiguoxin]

Duplicated by #4940