Async task state tracking

[nathanielrindlaub]

We now have a growing number of potentially long-running tasks that Lambdas are not well suited to support, especially if the user is expecting a synchronous response. These include:

• Bulk upload
• Deleting large numbers of images
• Deleting large numbers of labels (high priority, very expensive & hard to optimize with bulkWrite)
• Creating large numbers of labels (currently can handle ~15k new labels w/o async)
• Updating camera serial numbers
• Getting View stats
• Exporting annotations
• Updating deployments

@ingalls recommends that we consider creating a consistent pattern for tracking these async tasks in the DB (creating a collection or collections that we update when the state of one of these processes changes and a consistent query pattern for accessing them). I think it's a great idea... right now we have 2 entirely different ways of checking the bulk upload state and the annotation state, and for most of the others we haven't yet implemented spinning the tasks off on separate infrastructure; instead, we only support them at pretty low thresholds.

For the tasks we haven't yet broken out to run async, we have a couple options:

1. Using AWS Batch (same process as batch upload) w/ Fargate. This would be more expensive and have a much longer cold-start, but it would mean that we wouldn’t be limited to Lambda’s 15 min limit
2. Create SQSs + Lambda workers to pull messages off and process them in separate Lambdas. Nick's main concern here is how do we make the UX make sense for tasks that run longer than 15 mins (and would require re-prompting/initiating by the user)

[postfalk]

I think that is a good idea. I have a similar design with my Lens to Salesforce workflow.

[nathanielrindlaub]

@ingalls, I was just made aware that the process of creating, updating, or deleting deployments on cameras with large numbers of images (like 20k) will timeout, and the deployment update will fail. This is because ProjectModel.reMapImagesToDeps() has to iterate over every image in the camera to make sure it's assigned to the right deployment.

So once we get the task tracking in place I think our two priorities should be:

1. making getStats async
2. making reMapImagesToDeps (or all deployment CRUD ops) async

[nathanielrindlaub]

I think this is just about done. We've migrated getStats, exportData, exportImageErrors, andcreate/update/deleteDeployments to the new async task lambda. We decided against using the task pattern for batches as they are much more involved and deeply integrated in the rest of the code base. It's worth, however, looking at whether we can now move some of the other time-consuming operations (Deleting large numbers of images, Deleting large numbers of labels, Merging large numbers of labels) to async and increasing or removing the allowable operation threshold.

Final punchlist:

• review access patterns / interfaces between new task handler the various DB Models they interface with. Right now there's some inconsistency and semi-confusing function names (see: Async UpdateDeployment #168 (comment))
• review frontend task slice and see if there are any opportunities to streamline or abstract it. At the very least, generalize getTaskFailure action handler so it works for all task types.