Job status websocket / websocket storm updates

[jakemcdermott]

SUMMARY

We're going to add some new behavior to the websocket subscription flow in the API and UI. The goal is to address the "websocket storm" issue.

details

[ryanpetrello]

@mabashian @jakemcdermott @rebeccahhh re the job list performance issue, I think we can make a lot of progress by just being smarter about the requests we make.

Here's an m5.4xl w/ 1M jobs and 1B+ job events:

Note if we drill down only to job IDs we care about (perhaps what just got sent over the websocket w/ a status change), things really get snappy: