PROD-2644 Improve logging and error visibility for TraversalErrors

[erosselli]

Closes PROD-2644

Description Of Changes

When a TraversalError is raised as part of running an access or erasure request the logged error does not include the privacy request id, making it hard to find, and the UI shows the request in an error state but provides no information on what the error actually was. This PR adds a log that includes the privacy request id, and also creates an ExecutionLog instance so the error can be surfaced in the UI.

Code Changes

• Created two specific error classes UnreachableNodeError and UnreachableEdgeError that inherit from TraversalError, and throw those instead when we have an unreachable node or edge, respectively.
• Catch all TraversalErrors in both run_access_request and run_access_request_deprecated , log the error including the id of the privacy request , create the necessary ExecutionLog instances, and raise the error again.
• Unrelated to the ticket but had to remove the with_server_health_check decorator from both the worker and status commands, they were mistakenly added in a recent PR.

Steps to Confirm

Setup

• Start up Fides with the postgres integration, i.e nox -s dev -- postgres
• Create two datasets that have no identities and no references, such as

    fides_key: prod-2644-example
    name: Example PROD-2644
    description: Example to reproduce PROD-2644
    collections:
      - name: address
        fields:
          - name: city
            data_categories: [user.contact.address.city]
          - name: house
            data_categories: [user.contact.address.street]

and

    fides_key: prod-2644-example-2
    name: Example PROD-2644 2
    description: Second example to reproduce PROD-2644
    collections:
      - name: login
        fields:
          - name: id
            data_categories: [user.unique_id]
          - name: customer_id
            data_categories: [user.unique_id]
      - name: report
        fields:
          - name: id
            data_categories: [user.unique_id]
          - name: email
            data_categories: [user.contact.email]

• Add two different Postgres integrations to connect to the postgres DB running on port 6432
• For each integration, go to the integration details, and choose one of the datasets you just created (e.g prod-2644-example) from the Dataset dropdown. You should have two integrations, each linked to a different dataset

DSR 2.0

• From the Privacy Center or the Swagger API , create a Privacy Request.
• In the Admin UI, open the created privacy request: it should be in error status, and there should be an Error log in the Activity Timeline section (see screenshots below)

You should also see the log in the fides container, e.g
2024-09-06 13:57:08 2024-09-06 16:57:08.400 | ERROR | fides.api.task.deprecated_graph_task:run_access_request_deprecated:116 - TraversalError encountered for privacy request pri_83ace5f2-e4f7-4c2b-9ce3-fb989cddb214. Error: Some nodes were not reachable: prod-2644-example:address

DSR 3.0

• Stop your fides server
• Open the fides.toml file, and replace the following two lines:
  • in the [execution] section, change use_dsr_3_0=false to use_dsr_3_0=true
  • in the [celery] section, change task_always_eager = true to task_always_eager = false
• Run the server with nox -s dev -- shell postgres worker
• Make sure the integrations and datasets you created in the Setup section still exist.
• Repeat the same steps as for DSR 2.0 -- you may need to manually approve the request from the Admin UI, and wait a bit for the task to run in the worker.

Other error cases

When the error is not for unreachable nodes, but rather a more generic TraversalError (e.g there's a cycle) then the error's shown a bit differently in the UI

Pre-Merge Checklist

• All CI Pipelines Succeeded
• Documentation:
  • documentation complete, PR opened in fidesdocs
  • documentation issue created in fidesdocs
  • if there are any new client scopes created as part of the pull request, remember to update public-facing documentation that references our scope registry
• Issue Requirements are Met
• Relevant Follow-Up Issues Created
• Update CHANGELOG.md
• For API changes, the Postman collection has been updated
• If there are any database migrations:
  • Ensure that your downrev is up to date with the latest revision on main
  • Ensure that your downgrade() migration is correct and works
    • If a downgrade migration is not possible for this change, please call this out in the PR description!

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
fides-plus-nightly	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Sep 12, 2024 6:02pm

[cypress]

fides    Run #9930

Run Properties:   Passed #9930  •  3a60219df3 ℹ️: Merge 343fe9dca2be49c04af424b64392c85c1769384c into aa37211d487f7ec3a5b38b58f923...

Project	fides
Branch Review	refs/pull/5263/merge
Run status	Passed #9930
Run duration	00m 36s
Commit	3a60219df3 ℹ️: Merge 343fe9dca2be49c04af424b64392c85c1769384c into aa37211d487f7ec3a5b38b58f923...
Committer	erosselli
View all properties for this run ↗︎

---

Test results
Failures	0
Flaky	0
Pending	0
Skipped	0
Passing	4
⚠️ You've recorded test results over your free plan limit. Upgrade your plan to view test results.
View all changes introduced in this branch ↗︎

[pattisdr]

@erosselli good start, also could you add some tests please?

[pattisdr]

great improvement, traversal errors have been something that has been buried for a long time - I think test coverage is good enough here. The "edges" case is not going to be hit anyway. Just a quick request to dry some of the code up -

[pattisdr]

Looks good, approved with passing tests

[cypress]

fides    Run #9937

Run Properties:   Passed #9937  •  07e53f4047: PROD-2644 Improve logging and error visibility for TraversalErrors (#5263)

Project	fides
Branch Review	main
Run status	Passed #9937
Run duration	00m 36s
Commit	07e53f4047: PROD-2644 Improve logging and error visibility for TraversalErrors (#5263)
Committer	erosselli
View all properties for this run ↗︎

---

Test results
Failures	0
Flaky	0
Pending	0
Skipped	0
Passing	4
⚠️ You've recorded test results over your free plan limit. Upgrade your plan to view test results.
View all changes introduced in this branch ↗︎