PAI become unavailable when users open too many job detail-pages

[Binyang2014]

Currently, PAI job-detail page will auto-refresh every 10 seconds. When user submit too many job instances, every refresh will take a long time to retrieve data. And this will block other requests send to rest-server

Options to solve this problem:

1. Disable auto-refresh in job detail page
2. Add cache in rest-server
3. Add PAI service alert. When PAI become unavailable, rest-server will consume more than 4GB memory and cpu usage is over 100%. Maybe we can trigger an alert for this. (Collecting rest-server logs will also helpful)

[fanyangCS]

In the short term, I suggest to:

• do not retrieve GPU index when querying job details
• Disable or increase the interval of auto-refresh when number of containers is larger than 64
• Exponential back-off auto refresh when API call is failing