Add fault tolerance support for trial failure

[jinan-zhou]

In practice, even the training container is correctly implemented, trials may fail due to a variety of reasons such as insufficient GPU memories. If trials fail, GetEvaluationResult() will return None. The suggestion should be able to handle that exception. My solution is:

• If not all the trials failed, use the metrics of the successful ones to do the update
• If all the trials failed:
  1. Firstly, try to respawn the previous trials after sleeping for RESPAWN_SLEEP seconds.
  2. If respawning the trials for RESPAWN_LIMIT times still cannot collect valid results, then fail the task because it may indicate that the training container has errors.

Besides, this PR also fixes some small bugs and adds some important TODOs.

An example of fault handling output

---------------------------------------------------------------------------
Suggestion Step 1 for StudyJob nas-example-1 (ID: e9850c4a885acb19)
---------------------------------------------------------------------------
>>> 2 Trials succeeded, 1 Trials failed:
p55d6ef3720fec95: Failed
n8f94938e123b38d: 0.6475
g30851e3c259d07a: 0.6709
The average is 0.6592

>>> Suggestion updated. LSTM Controller Reward: 44.43968200683594

---

This change is 

[jinan-zhou]

/assign @hougangliu

[andreyvelich]

@hougangliu @YujiOshima @johnugeorge
Do we have any way to make StudyJob Failed from Suggestion ?

[andreyvelich]

/lgtm

[jinan-zhou]

@hougangliu This PR is well tested and ready to merge. Please take a look.

[hougangliu]

/lgtm

[jinan-zhou]

Could you approve it so that I can go on @hougangliu

[hougangliu]

/approve
thanks!

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: hougangliu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [hougangliu]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment