Pause Erasure Execution for Manual Confirmation [#522]

[pattisdr]

➡️ Follow-up to #554. Follows same pattern, but for an erasure.

Purpose

Some data cannot be automatically masked or destroyed. During execution of a privacy request with configured erasures, pause on collections marked as manual and wait for the user to confirm that data has been removed or manually masked before proceeding.

Resume a paused erasure request by passing in the masked count for the paused collection:
POST {{host}}/privacy-request/{{privacy_request_id}}/erasure_confirm

{
    "row_count": 5
}

This mirrors what we return in an automated way when masking collections.

Changes

Big picture, changes added to store which nodes we've run erasures on as we run them, and separately to store which node we're currently paused on, as well as store manually-confirmed erasure counts. Finally, an endpoint is added to resume the privacy request from an erasure stage, more details below:

• Add an API endpoint to resume a privacy request that is paused at the erasure stage
  • Dry up the logic to share between this and an endpoint that is paused at the access stage. These are separate endpoints because a single privacy request can run both access and erasures
• Update graph_task.erasure_request to cache the row count erased for each collection (similar to how we currently cache data retrieved for access requests). The primary purpose of this is to track which collections we've already erased so the new graph doesn't visit them when we resume.
• Add update_erasure_mapping_from_cache which modifies the graph if we need to restart to avoid running erasures on the collections we've already visited. This is how "pausing" is effectively handled - we don't actually pause, but we force execution to stop when we hit a paused node, and then rebuild a different graph that just runs the remaining nodes on resume.
• Like access requests, force erasure tasks to run sequentially. With Pause Access Request and Resume with Manual Input [#521] #554, A "Pause" now throws an exception and cancels all other tasks in the graph. Running one at a time makes sure we don't close the session prematurely and affect other tasks in flight, potentially preventing execution logs from getting persisted for each collection. There are other ways to do this, but we decided in Pause Access Request and Resume with Manual Input [#521] #554 that this was a good path forward for now, seeing as we're already using Dask Delayed to execute the graph - its side effect is that an exception cancels all other tasks in the graph.
• Separately add methods to the PrivacyRequest to both cache manual erasure confirmations and retrieve them. These are used both when we resume a privacy request manually to see if we have the information needed to proceed.
• Add ManualConnector.mask_data which looks to see if a manual erasure confirmation has been added to the cache. Otherwise, it Pauses the graph. When we rerun after the user adds the manual confirmation via the API, it finds that and proceeds with execution.

Checklist

• Update CHANGELOG.md file
  • Merge in main so the most recent CHANGELOG.md file is being appended to
  • Add description within the Unreleased section in an appropriate category. Add a new category from the list at the top of the file if the needed one isn't already there.
  • Add a link to this PR at the end of the description with the PR number as the text. example: #1
• Applicable documentation updated (guides, quickstart, postman collections, tutorial, fidesdemo, database diagram.
• If docs updated (select one):
  • documentation complete, or draft/outline provided (tag docs-team to complete/review on this branch)
  • documentation issue created (tag docs-team to complete issue separately)
• Good unit test/integration test coverage
• This PR contains a DB migration. If checked, the reviewer should confirm with the author that the down_revision correctly references the previous migration before merging
• The Run Unsafe PR Checks label has been applied, and checks have passed, if this PR touches any external services

Worth noting

• We have a separate ticket to surface to the user how to perform this manual request, for both access and erasure. For both access and erasure requests, the user needs to know the paused collection, and how to locate the record in question (this would be whatever data is passed to the manual node from upstream collections). In addition, they might need to be given how to mask the record in question. Surface to user how to Pause/Resume Privacy Request  #570
• The current design makes erasures with manual data a two-step process (as erasures actually run both an access and an erasure). First, the user will need to retrieve the record in question and enter in specific fields. Second, the user will need to manually mask or destroy that data and send in a confirmation. If the data in question is something like a folder in a filing cabinet that needs to be destroyed, this feels like overkill. On the other hand, the file in the filing cabinet might be needed to find data in some other collection, so the user needs to pass in its contents. Additionally, the data may have more specific masking requirements beyond just destruction, hence the two-step process. Further product refinement may be needed here.
  • This ticket doesn't surface to the user how to manually mask the data, that will be included in Surface to user how to Pause/Resume Privacy Request  #570

Ticket

Fixes #522

[pattisdr]

@ethyca/docs-authors minor docs change added here to explain how to resume a privacy request that is paused in the erasure step.

[pattisdr]

I think I've responded to all your comments Paul, ready for another pass!

[sanders41]

lgtm

I see you have a note to the docs team so I didn't merge in case you still have something that needs looking at there.

[pattisdr]

Thank you @sanders41, i think docs team is out this week, so I'd go ahead and merge and I can make a note to him about the followup.