Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

LJ-453 #5834

Merged
merged 8 commits into from
Mar 4, 2025
Merged

LJ-453 #5834

merged 8 commits into from
Mar 4, 2025

Conversation

erosselli
Copy link
Contributor

@erosselli erosselli commented Mar 3, 2025

Closes LJ-453

Description Of Changes

Because some privacy requests can take very long to process (e.g 40min) , by the time some access tasks complete, the database session they were using has been closed and so we get SSL / transaction errors when they try to update their own status. Since they throw before they can save their status in the DB, these tasks get re-processed indefinitely when re-submitting the request.

This PR fixes this by using a brand new DB session every time GrpahTask.update_status gets called.

Code Changes

  • Moved get_messaging_service, get_privacy_request_service, get_dataset_service and get_dataset_config_service from src/fides/api/api/deps.py into their own file src/fides/api/service/deps.py to avoid circular imports
  • Updated GraphTask methods update_status and log_end to use new DB sessions

Steps to Confirm

  1. Update your fides.toml to run DSR 3.0 by setting use_dsr_3_0 = true under [execution]
  2. Create a System (or choose an existing one) and link an integration of your choosing to it (e.g Postgres )
  3. Create a Dataset for that integration and link it to the integration
  4. Run a DSR
  5. Privacy Request should run OK
  6. Change the credentials for the integration so that they are wrong (i.e integration will fail)
  7. Run another DSR
  8. Privacy Request should show in errored status as expected, with the Activity Timeline showing correctly
  9. Update your fides.toml back to DSR 2.0 , and re-run steps 2-8

Pre-Merge Checklist

  • Issue requirements met
  • All CI pipelines succeeded
  • CHANGELOG.md updated
    • Add a db-migration This indicates that a change includes a database migration label to the entry if your change includes a DB migration
    • Add a high-risk This issue suggests changes that have a high-probability of breaking existing code label to the entry if your change includes a high-risk change (i.e. potential for performance impact or unexpected regression) that should be flagged
  • Followup issues:
    • Followup issues created (include link)
    • No followup issues
  • Database migrations:
    • Ensure that your downrev is up to date with the latest revision on main
    • Ensure that your downgrade() migration is correct and works
      • If a downgrade migration is not possible for this change, please call this out in the PR description!
    • No migrations
  • Documentation:
    • Documentation complete, PR opened in fidesdocs
    • Documentation issue created in fidesdocs
    • If there are any new client scopes created as part of the pull request, remember to update public-facing documentation that references our scope registry
    • No documentation updates required

Copy link

vercel bot commented Mar 3, 2025

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
fides-privacy-center ✅ Ready (Inspect) Visit Preview 💬 Add feedback Mar 4, 2025 9:20pm
1 Skipped Deployment
Name Status Preview Comments Updated (UTC)
fides-plus-nightly ⬜️ Ignored (Inspect) Visit Preview Mar 4, 2025 9:20pm

Copy link
Contributor

@galvana galvana left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good, we should make sure we ticket the quirks around TaskResources and GraphTask session handling

@erosselli erosselli added the run unsafe ci checks Runs fides-related CI checks that require sensitive credentials label Mar 4, 2025
Copy link

codecov bot commented Mar 4, 2025

Codecov Report

Attention: Patch coverage is 95.23810% with 2 lines in your changes missing coverage. Please review.

Project coverage is 86.96%. Comparing base (e8893c7) to head (f13ca43).
Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
src/fides/api/task/graph_task.py 89.47% 1 Missing and 1 partial ⚠️

❌ Your patch status has failed because the patch coverage (95.23%) is below the target coverage (100.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #5834      +/-   ##
==========================================
+ Coverage   86.62%   86.96%   +0.33%     
==========================================
  Files         406      407       +1     
  Lines       25017    25038      +21     
  Branches     2695     2696       +1     
==========================================
+ Hits        21672    21775     +103     
+ Misses       2771     2680      -91     
- Partials      574      583       +9     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@erosselli erosselli merged commit ae2921e into main Mar 4, 2025
40 of 41 checks passed
@erosselli erosselli deleted the LJ-453 branch March 4, 2025 22:24
Copy link

cypress bot commented Mar 4, 2025

fides    Run #12625

Run Properties:  status check passed Passed #12625  •  git commit ae2921e304: LJ-435 Fix use of stale sessions in privacy request task processing (#5834)
Project fides
Branch Review main
Run status status check passed Passed #12625
Run duration 00m 50s
Commit git commit ae2921e304: LJ-435 Fix use of stale sessions in privacy request task processing (#5834)
Committer erosselli
View all properties for this run ↗︎

Test results
Tests that failed  Failures 0
Tests that were flaky  Flaky 0
Tests that did not run due to a developer annotating a test with .skip  Pending 0
Tests that did not run due to a failure in a mocha hook  Skipped 0
Tests that passed  Passing 5
View all changes introduced in this branch ↗︎

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
run unsafe ci checks Runs fides-related CI checks that require sensitive credentials
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants