Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create a short version of the workgraph data for the connectivity analysis #291

Merged
merged 2 commits into from
Sep 5, 2024

Conversation

superstar54
Copy link
Member

@superstar54 superstar54 commented Sep 5, 2024

When analyzing the connectivity of the graph, The actual input data of the tasks is not needed. Therefore, the ConnectivityAnalysis class from node_graph tries to deepcopy to create a short version. However, it raised an error when deepcoping of the orm.computer, as reported in #290 .

In principle, we can fix this in ConnectivityAnalysis in node_graph, but we can also fix it in workgraph. This PR tries to create the short version data before passing to ConnectivityAnalysis, and avoids deepcopy the data of the tasks but only picks the info that is needed.
This is fixed in the node-graph package, and this PR bumped node-graph to the latest version.

@superstar54 superstar54 changed the title Create a short version of the workgraph data for the connectivity ana… Create a short version of the workgraph data for the connectivity analysis Sep 5, 2024
@superstar54 superstar54 linked an issue Sep 5, 2024 that may be closed by this pull request
This avoid deepcopy the input data of the tasks
@superstar54 superstar54 force-pushed the fix/290/avolid_deepcopy_orm_data branch from 9df798f to 4aef50d Compare September 5, 2024 10:17
@codecov-commenter
Copy link

codecov-commenter commented Sep 5, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 79.81%. Comparing base (5937b88) to head (4a1de1f).
Report is 60 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #291      +/-   ##
==========================================
+ Coverage   75.75%   79.81%   +4.06%     
==========================================
  Files          70       65       -5     
  Lines        4615     4925     +310     
==========================================
+ Hits         3496     3931     +435     
+ Misses       1119      994     -125     
Flag Coverage Δ
python-3.11 79.73% <100.00%> (+4.06%) ⬆️
python-3.12 79.73% <100.00%> (?)
python-3.9 79.77% <100.00%> (+4.03%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Contributor

@agoscinski agoscinski left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks fixes it for me.

@@ -294,6 +291,9 @@ def check_diff(
from node_graph.analysis import DifferenceAnalysis

wg1 = self.get_wgdata_from_db(restart_process)
# also make a alias for nodes
wg1["nodes"] = wg1["tasks"]
self.wgdata["nodes"] = self.wgdata["tasks"]
Copy link
Contributor

@agoscinski agoscinski Sep 5, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why do you not pop the item here as in the code changes below?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Keeping both tasks and nodes will not raise an error in DifferenceAnalysis. But for consistency, I changed it to be the same as the code below.

@superstar54 superstar54 merged commit cae7fed into main Sep 5, 2024
8 checks passed
@superstar54 superstar54 deleted the fix/290/avolid_deepcopy_orm_data branch September 5, 2024 16:12
@GeigerJ2
Copy link
Contributor

GeigerJ2 commented Sep 5, 2024

@superstar54 you are too quick :D was about to report that I had the same error today, and this also fixes it for me!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Setting computer in ShellJob raises pickle error
4 participants