[BUG] Can't get or delete workflow state without template if filtering by user

[dbwiddis]

What is the bug?

In most circumstances, we pair a workflow state document with a template document, sharing the same document id.

Our API permits users to delete the template while keeping the state document (by default; users can specify to clear status to delete both).

However, when access control is enabled, only the template contains the User that can be used to validate permission for various APIs including:

• Provision workflow
• Reprovision workflow
• Get workflow
• Delete workflow
• Deprovision workflow
• Get workflow state

For the Provision, Reprovision, and Get workflow cases, failure is expected if the template has been deleted. However, for the other three APIs, the lack of template means there is no way to even view, or delete the state document:

• Delete workflow may not seem needed since the template is already deleted, but this is one way to delete the state document using the clear status parameter
• Deprovisioning deletes the state on success if the template doesn't exist, but you can't even execute deprovisioning
• Get workflow state has no concept of the user so no check can be conducted

In summary, we have an orphaned document that the user can't either view or delete.

How can one reproduce the bug?

1. Enable filtering by user
2. Create and provision a workflow
3. Delete the workflow without specifying clear_status=true
4. Attempt any of the above APIs
   • Delete workflow with clear_status=true fails as access can't be checked
   • Deprovision workflow fails as as access can't be checked
   • Get workflow state fails as access can't be checked

What is the expected behavior?

User can clean up all documents in their cluster without direct access to system indices

Do you have any additional context?

Discovered this in the course of integ tests for #980, which adds additional complexity of tenant id validation. As tenant id is also added to the workflow state document, it's possible to work around it by falling back to tenant id check of workflow state in case of a template 404. This suggests we could/should also add the User to the workflow state document in order to permit the same fallbacks.