SanityBot

[marcua]

Problem

When experts are working on tens of projects at once, it's easy to miss a step or lose situational awareness on a particular project.

Solution

A SanityBot that periodically looks at the state of a project and reminds the project team about various things that seem off.

Use cases

These use cases are heavily biased by the design delivery process at B12. While projects are more complex, we'll consider a project with a customer success manager (CSM) and a designer.

• A CSM has been on a project for some time but has neither completed setting up the project or asked the customer for clarifying information.
• The project is set up, but a designer has not been staffed to the project for some time.
• A designer is staffed to a project, but we haven't sent v1 of a design to a customer in some time.
• We've sent the v1 of a design to a customer, but haven't received confirmation/started iterating with the customer in some time.
• We've been iterating with a customer for some time with no result.
• We've done more than X hours of work on a project.
• A todo hasn't been checked off in some time.

Other features that are off the table for now

• Projects/tasks should have "todo scaffolds," which are collections of todos that get created at various points in a project's lifetime.
• Todos should have due dates, and should be snoozable for some amount of time before they pop up (reminders). If Todos could be set as reminders in some time, you could imagine not warning project teams about tasks while there is a snoozed Todo.
• It would be nice to have a view of all of the Todos assigned to you across all projects, rather than having to click into individual tasks.

How to surface sanity checks

• Send a message to the project slack channel. Quick user research indicates that this is CSMs' preferred modality for surfacing checks.
• Send a message to the internal (e.g., entire company) slack group/channel. This is for escalating sanity checks in cases where the project is very off track.
• Create TODOs for team members (CSMs) when a sanity check occurs
• Indicate tasks that have new sanity checks (Todos?) in the dashboard.

Implementation notes

• Create a SanityCheck model with the following fields
  • created_at: datetime
  • is_deleted: datetime
  • handled_at: datetime
  • project: foreign key
  • check_slug: a slug for the type of check that's fired. No uniqueness constraints on this---you might create multiple of the same sanity check every X hours/days.
• In version.json, add an optional sanity_checks key with the following schema

  ...
  "sanity_checks": {
  "sanity_check_function": {
                "path": "path.to.sanity.check.function"
  },
  "check_configurations": {
     "check_slug1": {
        "handlers": [{"type": "slack_project_channel", "message": "<message here>", "steps": ["step_slug1", ...]}],
        "repetition_seconds": 3600
     },
     ...
  },
  ...
  

  • sanity_check_function is called periodically and generates SanityCheck objects. The function prototype is def sanity_check_function(project: Project) -> List[SanityCheck]:.
  • check_configurations maps SanityCheck.check_slug values to a configuration, which consists of a list of handlers and a repetition interval.
  • in v1, the only handler will be slack_project_channel, which messages the team slack project, tagging the experts assigned to the tasks specified by in steps.
  • repetition_seconds contains the number of seconds to wait before re-issuing/re-handling a SanityCheck. If repetition_seconds does not appear in the map, that SanityCheck is not repeated.
• A function orchestra.bots.sanitybot.create_and_handle_sanity_checks calls sanity_check_function for every project that has incomplete tasks, and call the appropriate handler on the returned SanityChecks.